kaourantin.net

64-bits

2008-11-16T21:01:00.000-08:00

Today we announced the availability of the Adobe Flash Player browser plugin for x86_64 Linux distributions. It is a preview release which has known bugs but should be fit initial testing by the community.

With this release we are tackling one the most requested feature ever for the Linux version of the Flash Player, even before windowless mode support. My personal hope though is that the constant flood of complaints we get about this every day will finally come to an end.

I've shown the 64 bit version of the Linux version to the public a couple of months ago, at the Flashforward 2008 conference and I have spent time stabilizing it since then. Sadly we have received zero contributions to make the open source Tamarin VM work properly in 64-bit mode on Linux which would have possibly allowed us to get this done sooner. The good news for me personally is that I now have a thorough understanding of the different x86-64 calling conventions and the instruction set.

The areas which remain untested and non-functional are the following:

Camera
Microphone
Fullscreen playback using OpenGL
Various Flash Player 10 specific features

This initial version has no .rpm or .deb packages and is therefore intended for advanced users. We still have work to do to reduce the number of dependencies of the binary and handle situations where installing this plugin might conflict with packages like nspluginwrapper.

Talking about nspluginwrapper: I strongly suggest not to use it. I know that some distros are thinking of even wrapping 64-bit plugins including Ubuntu with the thought that it will improve security and stability of the browser. This is a very bad idea in the state nspluginwrapper is in today. We have done some internal testing and discovered that several features in the Flash Player are broken when the plugin is wrapped. More importantly performance and user experience is pretty bad when the plugin is wrapped. Why? Lots of data needs to be transfered through IPC channels. I hope that browser vendors will eventually come up with a better architecture to wrap plugins without sacrificing performance, stability and functionality.

Please do report bugs or other issues you find using the right channel. And that means our public bug database:

https://bugs.adobe.com/flashplayer/

Comments on blogs, other web sites or 3rd party bug databases are not tracked by our quality assurance team. You are welcome to cross reference when you submit bugs though.

Now for some random items:

All SIMD optimizations which where in the 32bit version have been ported. No exceptions.
The 64-bit version is not faster than the 32-bit version as the 32-bit version was already well optimized. Scott Byer explained why some time ago: 64 bits...when?. You will see a big difference though when you get rid of nspluginwrapper.
The first release of 64-bit Flash Player code was actually part of Adobe Photoshop Lightroom 64-bit in form of an authplay.dll which is a plugin for applications. Second one was with Adobe Photoshop CS4 64-bit, also as an authplay.dll.
The first 64-bit plugin for a browser we ship is this Linux version. Windows and Mac will come later.
The 64-bit version of the plugin compiles and runs on FreeBSD 7.0 which I demoed at Flashforward 2008. There are no plans for release yet as it is still rather unstable and will require substantial work to get it ready for public consumption.
A debugger version of the 64-bit version is not available yet. When we release it ActionScript 2 debugging will not work due the obsolete protocol which depends on 32bit pointers. ActionScript 3 debugging will be supported.

Audio mixing with Pixel Bender

2008-10-09T14:00:00.000-07:00

Time to have some advanced fun with Pixel Bender. Recently someone in the community complained to us that mixing 13 mp3 tracks using the dynamic sound playback feature in Flash Player 10 does not really work. Well, true with the sample project he gave us. Doing dynamic sound playback is generally tricky to get right. I can provide a few tips though.

1. Pick the right mp3 encoding format

It's important you pick a format which consumes the least amount of CPU time for decoding. Specifically you should always choose 44.1Khz as the sample rate for your mp3 files. Why? The Flash Player will otherwise have to re-sample and filter your audio which takes away precious CPU cycles.

The tricky part here is that mp3 encoders usually pick the sample rate automatically, including Adobe Audition. Especially at a bit rate of 64kb or less it will try to switch to 24Khz or 22Khz. You can override this at least for CBR in Audition using the advanced settings at export time if you need to.

2. Keep things simple

Do all your processing in one function if you can. Function calls are expensive generally. Try to read and write data only once. Ideally your mixing code should look something like this if you use pure ActionScript:

var buffer:Vector.<ByteArray> = new Vector.<ByteArray>(NUM_TRACKS);
var sound:Vector.<Sound> = new Vector.<Sound>(NUM_TRACKS);

function onSoundData(sampleDataEvent:SampleDataEvent) : void
{
  for (var i:int = 0; i <  NUM_TRACKS; i++) {
    buffer[i].position = 0;
    sound[i].extract(buffer[i], BUFFER_SIZE); 
    buffer[i].position = 0;
  }

  for (var j:int = 0; j < BUFFER_SIZE*2; j++) 
  {
    var val:Number = 0;
    for (var k:int = 0; k < NUM_TRACKS; k++) 
    {
      val += buffer[k].readFloat();
    }
    sampleDataEvent.data.writeFloat(val);
  }
}

You will notice that you will spend a lot of time in this function. So...

3. Use Pixel Bender to mix sounds

I have talked to some who have tried to use Pixel Bender for audio processing. They had little success most of the time. Truth is, our tools are not ready yet. But with some patience and using the assembler for creating .pbj files I posted recently you can make it happen today.

One problematic issue is that right now the Pixel Bender toolkit is designed to handle image data. What does that mean? The toolkit limits you to float3 and float4 output types right now which is not really what you want. Now you might think you could just use float4. Not so. You will notice that Flash Player 10 has a pretty bad bug which makes it not work when float4 types are used for output. I am really angry we did not catch this sooner, hopefully we can address this bug sooner than later. What you are left with is using pure Pixel Bender assembly code for now which allows you to use a float2 output type.

For my experiment I took the Adobe Audition theme sample project and exported all tracks as separate .mp3 files, 15 in total. Incidentally that is also the maximum amount of inputs you can use for a single shader. The goal was to mix all 15 tracks in real time using the dynamic sound playback feature.

Here is the Pixel Bender assembly code I used to create my .pbj file:


  version 1
  name "SoundMixer"
  kernel "namespace", "adobe"
  kernel "vendor", "Adobe Systems"
  kernel "version", 1
  kernel "description", "A generic sound mixer with volume control"

  parameter "_OutCoord", float2, f0.rg, in

  texture "track0", t0.rg
  texture "track1", t1.rg
  texture "track2", t2.rg
  texture "track3", t3.rg
  texture "track4", t4.rg
  texture "track5", t5.rg
  texture "track6", t6.rg
  texture "track7", t7.rg
  texture "track8", t8.rg
  texture "track9", t9.rg
  texture "track10", t10.rg
  texture "track11", t11.rg
  texture "track12", t12.rg
  texture "track13", t13.rg
  texture "track14", t14.rg

  parameter "volume0", float2, f3.rg, in
  meta "defaultValue", 1, 1
  parameter "volume1", float2, f4.rg, in
  meta "defaultValue", 1, 1
  parameter "volume2", float2, f5.rg, in
  meta "defaultValue", 1, 1
  parameter "volume3", float2, f6.rg, in
  meta "defaultValue", 1, 1
  parameter "volume4", float2, f7.rg, in
  meta "defaultValue", 1, 1
  parameter "volume5", float2, f8.rg, in
  meta "defaultValue", 1, 1
  parameter "volume6", float2, f9.rg, in
  meta "defaultValue", 1, 1
  parameter "volume7", float2, f10.rg, in
  meta "defaultValue", 1, 1
  parameter "volume8", float2, f11.rg, in
  meta "defaultValue", 1, 1
  parameter "volume9", float2, f12.rg, in
  meta "defaultValue", 1, 1
  parameter "volume10", float2, f13.rg, in
  meta "defaultValue", 1, 1
  parameter "volume11", float2, f14.rg, in
  meta "defaultValue", 1, 1
  parameter "volume12", float2, f15.rg, in
  meta "defaultValue", 1, 1
  parameter "volume13", float2, f16.rg, in
  meta "defaultValue", 1, 1
  parameter "volume14", float2, f17.rg, in
  meta "defaultValue", 1, 1

  parameter "output", float2, f1.rg, out

;----------------------------------------------------------

  texn f1.rg, f0.rg, t0
  mul f1.rg, f3.rg
  texn f2.rg, f0.rg, t1
  mul f2.rg, f4.rg
  add f1.rg, f2.rg
  texn f2.rg, f0.rg, t2
  mul f2.rg, f5.rg
  add f1.rg, f2.rg
  texn f2.rg, f0.rg, t3
  mul f2.rg, f6.rg
  add f1.rg, f2.rg
  texn f2.rg, f0.rg, t4
  mul f2.rg, f7.rg
  add f1.rg, f2.rg
  texn f2.rg, f0.rg, t5
  mul f2.rg, f8.rg
  add f1.rg, f2.rg
  texn f2.rg, f0.rg, t6
  mul f2.rg, f9.rg
  add f1.rg, f2.rg
  texn f2.rg, f0.rg, t7
  mul f2.rg, f10.rg
  add f1.rg, f2.rg
  texn f2.rg, f0.rg, t8
  mul f2.rg, f11.rg
  add f1.rg, f2.rg
  texn f2.rg, f0.rg, t9
  mul f2.rg, f12.rg
  add f1.rg, f2.rg
  texn f2.rg, f0.rg, t10
  mul f2.rg, f13.rg
  add f1.rg, f2.rg
  texn f2.rg, f0.rg, t11
  mul f2.rg, f14.rg
  add f1.rg, f2.rg
  texn f2.rg, f0.rg, t12
  mul f2.rg, f15.rg
  add f1.rg, f2.rg
  texn f2.rg, f0.rg, t13
  mul f2.rg, f16.rg
  add f1.rg, f2.rg
  texn f2.rg, f0.rg, t14
  mul f2.rg, f17.rg
  add f1.rg, f2.rg

Looks complicated, but in fact this does nothing more that the above ActionScript code, with unrolled loops. As an extra you can control the volume on each track.

To use the shader I wrote this little piece (note that this is incomplete code, it will not compile):


// Create shader
[Embed(source="mixer.pbj", mimeType="application/octet-stream")]
var MixerShader:Class;
var mixerShader:Shader = new Shader(new MixerShader());

// buffers will become shader inputs
var buffer:Vector.<ByteArray> = new Vector.<ByteArray>(15);
// volume control volume on each track, 1.0 is full volume
var volume:Vector.<Number> = new Vector.<Number>(15);

// initialize the shader inputs and volume values
for (var j:int = 0; j < 15; j++) {
  volume[j]=1.0;
  buffer[j] = new ByteArray();
  // set so shader will always work even we have not enough tracks
  buffer[j].length = BUFFER_SIZE*4*2; 
  mixerShader.data["track"+j]["width"] = 1024;
  mixerShader.data["track"+j]["height"] = BUFFER_SIZE/1024;
  mixerShader.data["track"+j]["input"] = buffer[j];
}

function onSoundData(e:SampleDataEvent) : void
{
  // extract the mp3 data into our shader inputs
  for (var i:int = 0; i < NUM_TRACKS; i++) {
    buffer[i].position = 0;
    sounds[i].extract(buffer[i], BUFFER_SIZE); 
    buffer[i].position = 0;
  }
  // update the volume value in the shader
  for (var k:int = 0; k < NUM_TRACKS; k++)  {
    mixerShader.data["volume"+k]["value"] = [ volume[k], volume[k] ];
  }
  // mix!
  var mixerJob:ShaderJob = new ShaderJob(mixerShader, e.data, 1024, BUFFER_SIZE/1024);
  mixerJob.start(true);
}

Compared to the pure AS3 version this runs twice as fast overall. On my Core 2 Mac mixing the 15 tracks consumes about 24% of one CPU. So if you are doing simple audio mixing like this Pixel Bender is a good choice. YMMV depending on what application we are talking about and how much processing you need to do on the audio.

Pixel Bender .pbj files

2008-09-05T15:59:00.000-07:00

If you have been playing with Pixel Bender in Flash Player 10 you know the workflow:

Create your .pbk in the Pixel Bender Toolkit.
Export a .pbj binary from the Pixel Bender Toolkit.
Embed or dynamically load the .pbj file in your ActionScript.

There is still some mystery around .pbj files, i.e. the file format is neither documented nor is it clear what exactly it contains. While I can't offer documentation on the file format at this time (although that will happen eventually) what I can offer is an assembler and disassembler I quickly hacked together. I am mostly using for this my own debugging purposes. For those who want to tweak Pixel Bender in Flash to the max this is a really good way to go.

Please note that this is neither officially supported by Adobe nor do I guarantee any correctness or completeness of these two tools. The binary format could change at any time, you should not rely on it. I am really throwing this out to the world for educational purposes. There is no documentation on the syntax or format nor how to put together kernels from scratch. It's really up to you to make any sense out of it and I do not expect that any Adobe will ever use this syntax since I just made it up myself. Since these are a quite advanced tools I also assume you know how to compile C++ command line tools yourself ;-)

Enough disclaimers, here is the meat:

http://www.kaourantin.net/source/pbjtools/apbj.cpp
http://www.kaourantin.net/source/pbjtools/dpbj.cpp

But for the lazy I have pre-compiled two Windows binaries of these two command line tools. On OSX you can simply compile these using 'g++ apbj.cpp -o apbj' and 'g++ dpbj.cpp -o dpbj' if you have the developer tools installed on your system. Here are the Windows binaries:

http://www.kaourantin.net/source/pbjtools/apbj.zip
http://www.kaourantin.net/source/pbjtools/dpbj.zip

It should be fairly clear on how to use these if you run them in a command prompt. One of the goals was to allow perfect round tripping, i.e. disassemble->assemble->disassemble without any information loss. I hope I succeeded in this. I also know that someone has already put together an ActionScript version of this which is quite easily done.

Here is some sample output of the disassembler, the .pbj file was based on a Pixel Bender kernel created by Mr.doob (and btw, I fixed the bug he noticed in his blog entry ;-):


 version  1
 name  "NewFilter"
 kernel  "namespace", "Hypnotic"
 kernel  "vendor", "Mr.doob"
 kernel  "version", 1
 kernel  "description", "Hypnotic effect"

 parameter "_OutCoord", float2, f0.rg, in

 texture  "src", t0

 parameter "dst", float4, f1, out

 parameter "imgSize", float2, f0.ba, in
 meta  "defaultValue", 512, 512
 meta  "minValue", 0, 0
 meta  "maxValue", 512, 512

 parameter "center", float2, f2.rg, in
 meta  "defaultValue", 256, 256
 meta  "minValue", 0, 0
 meta  "maxValue", 512, 512

 parameter "offset", float2, f2.ba, in

;----------------------------------------------------------

 mov f3.rg, f0.rg
 sub f3.rg, f2.rg
 rcp f3.ba, f0.ba
 mul f3.ba, f3.rg
 mov f3.rg, f3.ba
 set f3.b, 3.14159
 mov f3.a, f3.g
 atan2 f3.a, f3.r
 mov f4.r, f3.a
 set f3.a, 2
 mov f4.g, f3.r
 pow f4.g, f3.a
 set f3.a, 2
 mov f4.b, f3.g
 pow f4.b, f3.a
 mov f3.a, f4.g
 add f3.a, f4.b
 sqr f4.g, f3.a
 mov f3.a, f4.g
 set f4.g, 0
 set f4.b, 0
 set f4.a, 0
 add f4.g, f2.b
 add f4.b, f2.a
 cos f5.r, f4.r
 rcp f5.g, f3.a
 mul f5.g, f5.r
 add f4.g, f5.g
 sin f5.r, f4.r
 rcp f5.g, f3.a
 mul f5.g, f5.r
 add f4.b, f5.g
 set f5.r, 1
 set f5.g, 0.1
 mov f5.b, f3.a
 pow f5.b, f5.g
 rcp f5.g, f5.b
 mul f5.g, f5.r
 add f4.a, f5.g
 mul f4.g, f0.b
 mul f4.b, f0.a
 set f5.r, 0
 ltn f4.g, f5.r
 mov i1.r, i0.r

 if i1.r

 set f5.r, 0
 sub f5.r, f4.g
 rcp f5.g, f0.b
 mul f5.g, f5.r
 ceil f5.r, f5.g
 mov f5.g, f0.b
 mul f5.g, f5.r
 add f4.g, f5.g

 end  

 set f5.r, 0
 ltn f4.b, f5.r
 mov i1.r, i0.r

 if i1.r

 set f5.r, 0
 sub f5.r, f4.b
 rcp f5.g, f0.a
 mul f5.g, f5.r
 ceil f5.r, f5.g
 mov f5.g, f0.a
 mul f5.g, f5.r
 add f4.b, f5.g

 end  

 ltn f0.b, f4.g
 mov i1.r, i0.r

 if i1.r

 rcp f5.r, f0.b
 mul f5.r, f4.g
 floor f5.g, f5.r
 mov f5.r, f0.b
 mul f5.r, f5.g
 sub f4.g, f5.r

 end  

 ltn f0.a, f4.b
 mov i1.r, i0.r

 if i1.r

 rcp f5.r, f0.a
 mul f5.r, f4.b
 floor f5.g, f5.r
 mov f5.r, f0.a
 mul f5.r, f5.g
 sub f4.b, f5.r

 end  

 mov f5.r, f4.g
 mov f5.g, f4.b
 texn f6, f5.rg, t0
 mov f1, f6
 mul f1.rgb, f4.aaa

On Performance

2008-09-03T15:43:00.001-07:00

With the release of Google Chrome I see blogs and articles blaming the Flash Player for poor performance and somehow linking this to the fact that it is not open source. Time to clarify a few bits. I'll start with classic comments:

"Flash hogs my CPU!"

1. HTML != Flash

HTML is a static document format. Flash (TM) content is in its core a classic multimedia format and most Flash content is still purely passive media.

What does that mean? When rendering HTML pages CPU usage only peaks for a very short of amount of time, essentially one single frame in Flash terms. After that almost no resources apart from memory are required. If you do not interact with the HTML page at all, no CPU time is required.

How does Flash compare? Most animated Flash content like rich media advertisement continues to use CPU resources to drive animation, video and/or sound. As opposed to static HTML which has exactly 1 frame, Flash content can have an infinite amount of frames which are played back over time.

Flash is great to provide experiences you could not get otherwise. Animation, video and sound are functions the browser does not (yet) provide, or at least they are not used to the same extend yet by designers. Once the browser will be used to play the same type of multimedia content you will face the same resource usage issues. It takes CPU cycles to decode video, sound and render animation. This is just a fact of life, we are however improving how much is used release after release, something benchmarks can back up.

So, there is a fundamental difference in media type. HTML is static, Flash is not. To put it in terms you might be able to understand:

If you take a picture and print it out you use energy only once and then can continue to view the picture forever without consuming any further energy. If you record a movie you will need some form of machine to play it back which will continue consume energy in form of a projector. The Flash Player is a projector.

"You are so full of it, AJAX does not hog my CPU!"

2. AJAX != Flash, but when done correctly AJAX can be the same as Flash

In most practical instances AJAX is used to drive RIAs. Examples include Gmail, Google Maps and many others. One fundamental property of good applications is that they only respond to either network activity or user input. Peak CPU usage is limited to these events. In general, if you do not touch the browser page no CPU time is required.

Compare this again to Flash animations, video and sound which in many cases remain passive experiences with no requirement for external events to drive the content. This will obviously use CPU resources continuously.

Now, it is perfectly possible to implement a Flash RIA application (that usually means using Flex) which uses the same or even less peak CPU than a AJAX RIA and only responds to network and user input. Flash is a flexible multiple paradigm platform, it depends on what the designer/programmer wanted to do. Unfortunately we at Adobe tend to see of a lot of RIAs which do not follow that principle and add lots of moving sugar to their applications which do little to improve actual usability.

Following good coding practices Flash can yield equivalent or better results than AJAX for many types of RIAs. Another benefit is that writing RIAs in Flash is truly cross browser as there is one Flash Player implementation only.

"Bull, SVG and Canvas show that it can be done better"

3. SVG/Canvas != Flash

Have you ever seen SVG or the canvas tag being used to implement anything else than static (1-frame) content? Have you ever seen rich media advertisement done using SVG or the Canvas tag? I mean not some demo page but actual deployed content. If so you will realize that the same resource usage issues apply.

"You are clueless, why does Flash suck up CPU time when it is on a hidden tab?"

4. Easy shortcuts do not work

Believe it or not but we and the browser vendors have tried to disable/pause/stop Flash content when a tab is hidden. The results were disastrous user experience wise to say the least. Disabling Flash to get any benefit CPU resource wise means the following:

Sound will have to stop
Any network transfer will have to stop
ActionScript execution will have to stop

Each one of these affect CPU resource usage and would affect user experience if we would turn it off. However the Flash Player does not render anything if it is on a hidden tab, we only execute the operations mentioned in the above list.

There is one exception to the rendering optimization: WMODE. If you use WMODE the Flash Player has no way of knowing if it is hidden or not and will continue to do a full render. Do not use WMODE. Unfortunately lots of rich media advertisement I see out there continues to enable this for no apparent reason.

"Flash sucks!"

5. You can help to educate web designers so common mistakes are not made

Huge help would be to adopt strict policies especially for rich media advertisement. I like the rules Google has put forward for Flash ads. Quoting:

"Animation Length: Animated ads are restricted to a maximum of 15 seconds (at a 15-20 fps frame rate), after which point they must remain static. These ads must also comply with the other animation policies."

Personally I would go even further and request the following:

After the animation has played no CPU resources should be used, ActionScript should be on a stop() command.
Mouse tracking or other event handling is not allowed unless you activate the banner with a mouse click.
DO NOT USE WMODE UNLESS YOU ABSOLUTELY NEED TRANSPARENCY! I can't stress that enough. Given the architecture of plugins there is no way for the Flash Player to know if Flash content is on a hidden tab or not and disable rendering properly. If you use WMODE the Flash Player will continue to suck up CPU cycles as if the tab was visible. In addition WMODE is much slower than the normal mode.

These simple rules would address almost all the complaints we hear about. Adobe has unfortunately only limited influence on what content gets deployed, in this case it is really up to the community to balk at the web sites putting up content which impacts user experience negatively.

--------------------

Like with any powerful technology it is easy to shoot yourself in the foot and with the ease of use of Flash that is unfortunately too common.

Despite of that we are working with all browser vendors to improve performance and user experience whenever possible. There are differences between browsers and our goal is to close this gap once and for all. We are for example looking forward to work together with Google to improve Flash performance in Google Chrome.

On our (Adobe) side we are also looking forward to improve Flash performance further. Flash Player 10 for instance is making the first steps towards hardware accelerated rendering which will provide a huge boost in rendering performance. On the scripting side Tamarin-tracing will improve scripting performance dramatically. This is work we share with the Mozilla foundation which will use the same core libraries under the TaceMonkey project. The latest benchmarks are quite remarkable.

Adobe Flash Player 10 pre-release refresh

2008-07-02T11:30:00.000-07:00

We just just released another pre-release of Flash Player 10 (build 10.0.1.525). Go get it here and make sure you read the release notes. As a reminder, as we are nearing the release it becomes increasingly difficult for us to address bugs, especially if they are not crashers. If you have backwards compatibility issues (and I almost guarantee you that there will be some which will affect your content) please report them here (registration required) or here (no registration required).

There have been numerous stability and performance improvements. The most important additions are support for WMODE=transparent and V4L2 cameras (which is still work in progress) on Linux which addresses two of the top 3 feature requests on this platform.

If you have followed GUIMark at all you will notice that this version of the player runs this benchmark substantially better on OSX than any previous Flash Player version. It should be up to 3 times faster. How will this affect you? Well, OSX device text rendering got a huge performance boost. If you use lots of device text you will see a big difference. I posted more details in a comment here

Finally the dynamic sound APIs have slightly changed as I announced previously. I will be updating my posts [1][2] later today.

Adobe Pixel Bender in Flash Player 10 Beta

2008-05-20T19:20:00.000-07:00

Lee Brimelow has posted a snippet of code showing how to use Adobe Pixel Bender kernels in the Flash Player 10. Time for me to go into details about this feature. As usual there are surprises and unexpected behavior this feature holds. I'll keep this post without any sample code, but I'll promise to show some samples soon.

A long time ago, back in Flash Player 8 days we had the idea of adding a generic way to do bitmap filters. Hard coding bitmap filters like we did for Flash Player 8 is not only not flexible, but has the burden of adding huge amounts of native code into the player and having to optimize it for each and every platform. The issue for us has always been how you would author such generic filters. Various ideas were floating around but in the end there was one sticking point: we had no language and no compiler. After Macromedia's merger with Adobe the Flash Player and the Adobe Pixel Bender team came together and we finally had what we needed: a language and a compiler.

The Pixel Bender runtime in the Flash Player is drastically different from what you find in the Adobe Pixel Bender Toolkit. The only connection the Flash Player has is the byte code which the toolkit does generate, it generates files with the .pbj extension. A .pbj file contains a binary representation of opcodes/instructions of your Pixel Bender kernel, much the same way a .swf contains ActionScript3 byte code. The byte code itself is designed to translate well into a number of different run times, but for this Flash Player release the focus was a software run time.

You heard right, software run time. Pixel Bender kernels do not run using any GPU functionality whatsoever in Flash Player 10.

Take a breath. :-)

Running filters on a GPU has a number of critical limitation. If we would have supported the GPU to render filters in this release we would have had to fall back to software in many cases. Even if you have the right hardware. And then there is the little issue that we only would have enabled this in the 'gpu' wmode. So it is critical to have a well performing software fallback; and I mean one which does not suck like some other frameworks which we have tried first (and which I will not mention by name). A good software implementation also means you can reach more customers which simply do not have the required hardware, which is probably 80-90% of the machines connected to the web out there. Lastly this is the only way we can guarantee somewhat consistent results across platforms. Although I have to point out that that you'll see differences which are the result of compromises to get better performance.

So why did we not just integrate what the Adobe Pixel Bender Toolkit does, which does support GPUs? First, we need to run on 99% of all the machines out there, down to a plain Pentium I with MMX support running at 400Mhz. Secondly, I would hate to see the Flash Player installer grow by 2 or 3 megabytes in download size. That's not what the Flash Player is about. The software implementation in Flash Player 10 as it stands now clocks in at about 35KB of compressed code. -- I am perfectly aware that some filters would get faster by an order of two magnitudes(!) on a GPU. We know that too well and for this release you will have to deal with this limitation. The important thing to take away here is: A kernel which runs well in the toolkit might not run well at all in the Flash Player.

But... I have more news you might not like. ;-) If you ever run a Pixel Bender filter on PowerPC based Mac you will see that it runs about 10 times slower than on an Intel based Mac. For this release we only had time to implement a JIT code engine for Intel based CPUs. On a PowerPC Mac Pixel Bender kernels will run in interpreted mode. I leave it up to you to make a judgment of how this will affect you. All I can say: Be careful when deploying content using Pixel Bender filters, know your viewers.

Now for some more technical details: the JIT for Pixel Bender filters in Flash Player 10 support various instructions sets, down to plain x87 floating point math and up to SSE2 for some operations like texture sampling which take the most amount of time usually. Given the nature of these filters working like shaders, i.e. being embarrassingly parallel, running Pixel Bender kernels scales linearly with amount of CPUs/cores you have on your machine. On an 8-core machine you will usually be limited by memory bandwidth. Here is a CPU readout on my MacPro when I run a filter on a large image (3872x2592 pixels):

There are 4 different ways of using Pixel Bender kernels in the Flash Player. Let me start with most obvious one and come down to the more interesting case:

Filters. Use a Pixel Bender kernel as a filter on any DisplayObject. Obvious.
Fill. Use a Pixel Bender kernel to define your own fill type. Want a fancy star shaped high quality gradient? A nice UV gradient? Animated fills? No problem.
Blend mode. Not happy with the built-in blend modes? Simply build your own.

What about the 4th? Well, as you can see the ones in the list are designed for graphics only. The last one is more powerful than that. Instead of targeting a specific graphics primitive in the Flash Player, you can target BitmapData objects, ByteArrays or Vectors. Not only that but if you use ByteArray or Vector the data you handle are 32-bit floating point numbers for each channel, unlike BitmapData which is limited to 8-bit unsigned integers per channel. In the end this means you can use Pixel Bender kernels to not only do graphics stuff, but generic number crunching. If you can accept the 32-bit floating point limitation.

This 4th way of using Pixel Bender kernels runs completely separate from your main ActionScript code. It runs in separate thread which allows you to keep your UI responsive even if a Pixel Bender kernel takes a very long time to complete. This works fairly similar to a URLLoader. You send a request with all the information, including your source data, output objects, parameters etc. and a while later an event is dispatched telling you that it is finished. This will be great for any application which wants to do heavy processing.

In my next post I show some concrete examples of how you would use these Pixel Bender kernels in these different scenarios. For now I'll let this information sink in.

What follows are a few random technical nuggets I noted in my specification when it comes to the implementation in the Flash player, highly technical but important to know if you are pushing the limits of this feature:

The internal RGB color space of the Flash Player is alpha pre-multiplied and that is what the Pixel Bender kernel gets.

Output color values are always clamped against the alpha. This is not the case when the output is a ByteArray or Vector.

The maximum native JIT code buffer size for a kernel is 32KB, if you hit this limit which can happen with complex filters the Flash Player falls back to interpreted mode mode like it does in all cases on PowerPC based machines.

You can freely mix linear and nearest sampling in your kernel.

Maximum coordinate range is 24bit, that means for values outside the range of -4194304..4194303 coordinates will wrap when you sample and not clamp correctly anymore.

The linear sampler does sample up to 8bit of sub pixel information, meaning you'll get a maximum of 256 steps. This is also the case if you sample from a ByteArray with floating point data.

Math functions apart from simple multiplication, division, addition and subtracting work slightly differently on different platforms, depending on the C-library implementation or CPU.

In the Flash Player 10 beta vecLib is used on OSX for math functions. Slightly different results on OSX are the result. This might change in the final release as the results could be too different to be acceptable. (This is at least one instance where something will be significantly faster on Mac than on PC)

The JIT does not do intelligent caching. In the case of fills that means that each new fill will create a new code section and rejit.

There are usually 4 separate JIT'd code sections which each handle different total pixel counts, from 1 pixel to 4 pixels at a time. This is required for anti-aliasing as the rasterizer works with single pixel buffers in this case.

When an if-else-endif statement is encountered, the JIT switches to scalar mode, i.e. the if-else-ending section will be expanded up to 4 times as scalar code. Anything outside of a if-else-endif block is still processed as vectors. It's best to move sampling outside of if statements if practical. The final write to the destination is always vectorized.

The total number of JIT'd code is limited by the virtual address space. Each code section reserves 128Kbytes (4*32KB) of virtual address space.

The first 4 pixels rendered of every instance of a shader is run in interpreted mode, the native code generation is done during that first run. You might get artifacts if you depend on limit values as the interpreted mode uses different math functions. If you are on a multicore system, every new span rendered will create a new instance of a shader, i.e. the code is JITd 8*4 times on a 8-core system. This way the JIT is completely without any locks.

What does GPU acceleration mean?

2008-05-16T17:50:00.000-07:00

The just released Adobe® Flash® Player 10 beta version includes two new window modes (wmode) which control how the Flash Player pushes its graphics to the screen.

Traditionally there have been 3 modes:

normal: In this mode we are using plain bitmap drawing functions to get our rasterized images to the screen. On Windows that means using BitBlt to get the image to the screen on OSX we are using CopyBits or Quartz2D if the browser supports it.

transparent: This mode tries to do alpha blending on top of the HTML page, i.e. whatever is below the SWF will show through. The alpha blending is usually fairly expensive CPU resource wise so it is advised not to use this mode in normal cases. In Internet Explorer this code path does actually not going through BitBlt, it is using a DirectDraw context provided by the browser into which we composite the SWF.

opaque: Somewhat esoteric, but it is essentially like transparent, i.e. it is using DirectDraw in Internet Explorer. But instead of compositing the Flash Player just overwrites whatever is in the background. This mode behaves like normal on OSX and Linux.

Now to the new modes:

direct: This mode tries to use the fastest path to screen, or direct path if you will. In most cases it will ignore whatever the browser would want to do to have things like overlapping HTML menus or such work. A typical use case for this mode is video playback. On Windows this mode is using DirectDraw or Direct3D on Vista, on OSX and Linux we are using OpenGL. Fidelity should not be affected when you use this mode.

gpu: This is fully fledged compositing (+some extras) using some functionality of the graphics card. Think of it being similar to what OSX and Vista do for their desktop managers, the content of windows (in flash language that means movie clips) is still rendered using software, but the result is composited using hardware. When possible we also scale video natively in the card. More and more parts of our software rasterizer might move to the GPU over the next few Flash Player versions, this is just a start. On Windows this mode uses Direct3D, on OSX and Linux we are using OpenGL.

Now to the tricky part, things which will cause endless confusion if not explained:

1. Just because the Flash Player is using the video card for rendering does not mean it will be faster. In the majority of cases your content will become slower.

...

Confused yet? Good, that means you have the same understanding what GPU support means that everyone else has.

Content has to be specifically designed to work well with GPU functionality. The software rasterizer in the Flash Player can optimize a lot of cases the GPU cannot optimize, you as the designer will have to be aware of what a GPU does and adapt your content accordingly. I realize this statement is useless unless we can provide guidance, something we can hopefully achieve in the not to distant future.

2. The hardware requirements for the GPU mode are stiff. You will need at least a DirectX 9 class card. We essentially have the exact same hardware requirements as Windows Vista with Aero Glass enabled. Aero Glass uses exact same hardware functionality we do. So if Aero Glass does not work well on your machine the Flash Player will likely not be able to run well either in GPU mode (but to clarify, you do NOT need Aero Glash for the GPU mode to work in the Flash Player, I am merely talking about hardware requirements here).

3. Pixel fidelity is not guaranteed when you use the GPU mode. You have to expect that content will look different on different machines, even colors might not match perfectly. This includes video. Future Flash Players will change the look of your content in this mode. We will try our best to limit the pain but please bear in mind that in many cases we have no control on this.

Here is an example, left shows it running using the new gpu mode, right using the normal mode. This a video which is 320x240 pixels large showing red text and as you notice the gpu mode arguably looks better as the hardware does UV blending:

The downside in this specific case is that the edges of the text are not as crisp anymore.

4. Frames rates will max out at the screen refresh rate. So whatever frame rate you set in your Flash movie is meaningless if it is higher than 60. This is the case for both the 'direct' and 'gpu' mode. In most cases you should end up at a frame rate of around 50-55 due to dropped frames which occur from time to time for various reasons.

5. Please do not blindly enable either new mode (gpu or direct) in your content. Creating a GPU based context in the browser is very expensive and will drain memory and CPU resources to the point where the browser will become unresponsive. It is usually best practice to limit yourself to one SWF per HTML page using these modes. The target should be content taking over most of the page and doing full frame changes like video. Never ever, ever enable this for banners. Plain Flex applications should not use these modes either if they are not doing full screen refreshes.

6. GPU functionality ties us together with the video card manufacturers and their drivers. Given that you can expect that a significant amount of customers will not be able to view your content if you enable this mode due to driver incompabilities, and various defects in the software stack.

Finally, this beta version of the Flash Player is not yet tuned for maximum performance in the gpu mode. We are making progress, but all the above points will still apply in the long term.

The 'direct' mode should never make your content slower, except in respect to point 4 I made. It should either not change anything or lower CPU consumption somewhat with very large content, i.e. something larger than 1024x768 pixels.

What we'd ask you to do though is to give this a test drive. This is completely new land for us and we expect to encounter lots of obstacles, meaning bugs.

Adobe Is Making Some Noise Part 3

2008-05-14T23:02:00.000-07:00

[Update: I have updated the code sample to match the API changes in build 10.0.1.525]

Along with the new event to drive dynamically generated audio playback there is one gaping hole: Where do you get source audio from? Sure, you can load raw audio from external sources through ByteArray but that is less than efficient. What you really need is a way to have sound assets you can access from your ActionScript code.

In Flash Player 10 code named Astro the Sound object will have one more method which is designed to work together with the "samplesCallback" event handler. It will extract raw sound data from an existing sound asset. That means any mp3 file you have in the library or load externally can be accessed and processed. Let's look at an actual code example. I am passing through audio from an existing sound object to another sound object doing the dynamic audio playback:


var mp3sound:Sound = new Sound();
var dynamicSound:Sound = new Sound();
var samples:ByteArray = new ByteArray();

function sampleData(event:SampleDataEvent):void {
  samples.position = 0;
  var len:Number = mp3sound.extract(samples,1777);
  if ( len < 1777 ) {
    // seamless loop
    len += mp3sound.extract(samples,1777-len,0);
  }
  samples.position = 0;
  for ( var c:int=0; c < len; c++ ) {
    var left:Number = samples.readFloat();
    var right:Number = samples.readFloat();
    event.data.writeFloat(left);
    event.data.writeFloat(right);
 }
}

private function loadComplete(event:Event):void {
 dynamicSound.addEventListener("sampleData",sampleData);
 dynamicSound.play();
}

mp3sound.addEventListener(Event.COMPLETE, loadCompleteMP3);
mp3sound.load(new URLRequest("sound.mp3"));

Notice the extract() call here. This function will extract raw sound data from any existing sound object. The format returned is always 44100Khz stereo, the number format is 32-bit floating point which you can read and write using ByteArray.readFloat and ByteArray.writeFloat. The floating point values are normalized between -1.0 and 1.0. Here is the full prototype of this function:


function extract(target:ByteArray,
           length:Number,
           startPosition:Number = -1 ):Number;

target: A ByteArray object in which the extracted sound samples should be placed.
length: The number of sound samples to extract. A sample contains both the left and right channels -- that is, two 32-bit floating point values.
startPosition: The sample at which extraction should begin. If you don't specify a value, the first call to extract() starts at the beginning of the sound; subsequent calls without a value for startPosition progress sequentially through the sound.
extract() returns the number of samples which could be retrieved. This might be less than the length you requested at the very end of a sound.

With what we provide in Flash Player 10 we hope that we are addressing the most pressing needs of what you want to do with sound. I will likely be just a matter of time until we'll see high level frameworks done by the community on the magnitude of something like Papervision3D. The next couple of years should be very interesting indeed when it comes to sound on the web.

What's missing? Unfortunately some features did not make it into Flash Player 10: Extracting audio data from a microphone and extracting audio from a NetStream object. We are aware that both features are highly desirable, but for various reasons it was not possible to make this happen in this release.

Adobe Is Making Some Noise Part 2

2008-05-14T23:01:00.000-07:00

[Update: I have updated the code sample to match the API changes in build 10.0.1.525]

The public beta of Adobe® Flash® Player 10, code named Astro has been released. When you read the release notes you'll notice a small sections talking about audio:

"Dynamic Sound Generation — Dynamic sound generation extends the Sound class to play back dynamically created audio content through the use of an event listener on the Sound object."

Yes, in Flash Player 10 you will be able to dynamically create audio. It's not an all powerful API, it is designed to provide a low level abstraction of the native sound driver, hence providing the most flexible platform to build your music applications on. The API has one big compromise which I can't address without large infrastructural changes and that is latency. Latency is horrible to the point where some applications will simply not be possible. To improve latency will require profound changes in the Flash Player which I will tackle for the next major revision. But for now this simple API will likely change the way you think about sound in the Flash Player.

Programming dynamic sound it is all about how quickly and consistently you can deliver the data to the sound card. Most sound cards work using a ring buffer, meaning you as the programmer push data into that ring buffer while the sound card feeds from it at the same time. The high level APIs to deal with this revolve around two concepts: 1. the device model 2. the interrupt model.

For model 1 we run in a loop (usually in a thread) and write sound data to the device. The write will block if the ring buffer is full. The loop continues until the sound ends. This is the most common method of playing back sound on Unix like systems like Linux.

In model 2 we have a function which is called by the system (usually from an interrupt on older systems) in which the applications fills part of the ring buffer. The callback function is called whenever the sound card hits a point where it runs low on samples in the ring buffer. In an OS without real threading this is usually the only way to make sound playback work. MacOS9 or older and Windows 98 or older were using this system and OSX continues to provide a way to do this in CoreAudio. As ActionScript has no threading it is advisable to use this model. We could use frames event to implement a loop, but that would represent an odd programming model.

Flash Player 10 code named Astro supports a new event on the Sound object: "samplesCallback". It will be dispatched on regular interval requesting more audio data. In the event callback function you will have to fill a given ByteArray (Sound.samplesCallbackData) with a certain amount of sound data. The amount is variable, from 512 samples to 8192 samples per event. That is something you decide on and is a balance between performance and latency in your application. The less data you provide per event the more overhead is spent in the Flash Player. The more data you provide the longer the latency for your application will be. If you just play continious audio we suggest to use the maximum amount of data per event as the difference in overall performance can be quite large.

I should note that this API will slightly change (names changes only mostly) in the final release of the Flash Player, this beta represents an older build. I'll update this post with new code once the API is finalized.

Now some real code, some of you on the beta program have seen it. Here is how you play a continuous sine wave in Flash Player 10 with the smallest amount of code:


  var sound:Sound = new Sound();
  function sineWavGenerator(event:SampleDataEvent):void {
    for ( var c:int=0; c<1234; c++ ) {
      var sample:Number = Math.sin(
               (Number(c+event.position)/Math.PI/2))*0.25;
      event.data.writeFloat(sample);
      event.data.writeFloat(sample);
    }
  }
  sound.addEventListener("sampleData",sineWavGenerator);
  sound.play();

That's it. That simple. You can't get any more low level or flexible than this. The sample above is simple, actual code would probably not call Math.sin() in the inner loop. You would rather prepare a ByteArray or Array outside and copy the data from there.

The sound format is fixed at a sample rate of 44100Hz, 2 channels (stereo) and using 32bit floating point normalized samples. This is currently the highest quality format possible within the Flash Player. We will be targeting a more flexible system in a future Flash Player. It was not possible to offer different samples rates and more or less channels in this version. If you need to resample you can either use pure ActionScript 3 or even Adobe Pixel Bender.

The SamplesCallbackEvent.position property which is passed in is the sample position, not the time, of the segment of audio which is being requested. You can convert this value to milliseconds by dividing it by 44.1.

Your event handler has to provide at least 512 samples each time it is dispatched, at most 8192. If you provide less the Flash Player makes the assumption that you reached the end of the sound, will play the remaining samples and dispatch a SOUND_COMPLETE event. If you provide more than 8192 an exception occurs. In the sample above I use 1234 to make it clear that it can be any value between 512 and 8192

The event will be called in real time. That means you can inject new audio data interactively. The key part to understand here that we are not dealing with long amounts of sound data at any given time.

There is an internal buffer in the Flash Player which is about 0.2 to 0.5 seconds depending on the platform which is preventing drop outs. It will automatically be increased if drop outs occur. This internal buffer is the key for the high latency I was alluding to earlier. You should never depend on a certain latency with this API in your application. To enforce this there is a slight random factor in choosing this buffer size when the Flash Player launches.

Continue to read Part 3 which talks about one more new Sound API in Flash Player 10.

Adobe Is Making Some Noise Part 1

2008-05-14T23:00:00.000-07:00

If you have been following the community there has been quite a stir around lack of sound features in the Flash Player. Not only lack of sound features, but the last two dot releases have injected serious bugs in the sound support. This is unfortunate and some of these are squarely my fault, but some of them are design choices for which there are no workarounds.

The big controversy has been around the SOUND_COMPLETE event which has become less predictable. There is a change I had to make for Windows Vista to fix an audio-video synchronization issue. Windows Vista uses a new audio stack and does support the older but still viable WaveOut API through an emulation layer. This emulation layer does not behave exactly like previous iterations of Windows and exposed some bad assumptions the Flash Player made to drive audio-video synchronization and the famous the SOUND_COMPLETE event.

To be clear: the accuracy of SOUND_COMPLETE will get even worse in Flash Player 10. Here, I said it. Why? Because the world moves on and power consumption of devices becomes more important than ever, even for desktops. One of the stumbling blocks of improving power consumption on Windows was the Flash Player and we've gotten an earful from Microsoft and Intel because of it. The key part here was a Win32 API we used which is timeBeginPeriod()/timeEndPeriod(). We were setting the period to 1ms so that events like SOUND_COMPLETE would be more accurate, dispatched at the right time. This worked on Win98 through WinXP but is essentially useless on Windows Vista since the sound stack works in a different way as I mentioned above. So apart from being useless on Vista using this API is deadly for power consumption and a show case of bad engineering. We can not continue to rely on it.

So what does that mean? You should have never relied on SOUND_COMPLETE in the first place to do sound processing. Easy for me to say, but Macromedia did miss putting a big warning into the documentation. Or maybe the real problem was that no one really paid attention to the sound stack for such a long time. I am trying to change this, unfortunately in the process causing havoc for some clients and breaking backwards compatibility.

Continue to read Part 2 to see how we will address the requests from the community in Flash Player 10.

Adobe Open Screen Project

2008-05-02T13:07:00.000-07:00

Unless you have been living under a rock you probably saw the announcement Adobe made this week:

http://www.adobe.com/openscreenproject/

This is a significant move by Adobe to bring the Adobe® Flash® Player to a much broader audience than ever before. By doing this we are responding to the requests from customers throughout the different industries which want to leverage the Flash platform.

So, yes I haven't been blogging in a while, that has been due to my busy schedule (I'll be more specific soon). But since I've worked on fixing up the FLV and SWF specs a little bit for this project I figured I should add my own personal thoughts.

As the site above mentions this project essentially consists of the following changes:

Removing restrictions on use of the SWF and FLV/F4V specifications
Publishing the device porting layer APIs for Adobe Flash Player
Publishing the Adobe Flash® Cast™ protocol and the AMF protocol for robust data services
Removing licensing fees – making next major releases of Adobe Flash Player and Adobe AIR for devices free

All of these are very significant for a lot of customers although individual parties might think that it is one particular item which matters most to them. I assure you that all of them are extremely important.

Bringing a platform like the Adobe Flash Player to devices has its challenges. One of them revolves around documentation. By making the file format and API documentation open, the entry barrier for a lot of developers becomes much lower. In the past it could have been a struggle to get the necessary documentation to the teams which need them. This problem is now gone.

In many circumstances technologies Adobe provided were not compatible with how a particular device or infrastructure was set up. Given the old licensing restrictions it could prove difficult to find technical solutions. By making the specifications open and without restrictions a particular vendor might instead choose a clean rewrite of some parts if they see fit with the result that it integrates better into their eco-system.

Now there have been questions if there are any gotchas in what Adobe has announced concerning the SWF and FLV specifications. There IS in fact a license, it is on page two of the specifications. You might be shocked however by how small it is. It's essentially BSD license style damages disclaimer plus some additional information about trademarks and such. Read it before you use it.

The specifications are available here as PDF files:

http://www.adobe.com/devnet/swf/
http://www.adobe.com/devnet/flv/

Does that mean you can implement your own clone which implements what is in these specifications without fear of Adobe coming after you? Yes. Notice that I am careful how I ask this question though. Adobe still holds all the trademarks around this technology, you can not use any of those in your clone or anything related to it. Unless we give the OK to do so obviously.

If you find inaccuracies or missing information in these specifications let us know, we can integrate the feedback right away so they can appear in the next revision of these documents.

Adobe Flash Player 9 Update 3 released

2007-12-03T21:09:00.000-08:00

We made it! We just shipped Adobe Flash Player 9 Update 3, the final version number is 9.0.115.0. As usual a number of different flavors of the web plugin are available, the main install page is here. And yes, we also shipped the Linux x86-32 version. We have done this for several versions now and we will not stop. The only exception is the Solaris version which is still in testing. I can't tell you though when a new player for Flash CS3 will be available since I do not know.

This version is a significant milestone, and I expect it will be adopted fairly quickly once content using H.264 and AAC starts appearing. Emmy Huang has some additional information and links you should look at.

Here is a condensed list of bugs which have been fixed since the last release candidate (9.0.64.0). As you can see these have been mostly fixes for crashers and backwards compatibility issues. It's a pretty long list which explains why 2 months have passed since the last release candidate. Maybe you'll spot a few you were affected by:

212379 audio playback in swf off sync.
212294 on (release) events not firing on sprites
212224 Heavy FMS Seeking in Vista can cause bad audio and crash
211921 Crash when paging through a Flex app and closing the player
211898 FileReference IO errors when uploading multiple files
211840 playlist does not advance after end of commercial
211836 [MP4] If you seek in the NetStream before a movie that has the moov atom at the end has completely downloaded, you can crash.
211813 BitmapData draw ColorTransform not affecting all text on all platforms
211759 inputted simplified Chinese characters are not correct inside of Flash Player
211727 Japanese/RHEL4/Firefox 1.0 -- hang when selecting print dialog
211725 Player crashes when Premiere Express is connected to a server that times out
211700 Add support for Opera XEmbed
211672 the pop-up print panel layout is incorrect when R-click on the red button and choose ‘print…’
211625 Flash player crashes while playing servierside playlists containing MP4 files
211593 Crash in NavigateToURL when using local relative URL
211584 can't input anything to flashplayer for Simplified Chinese Linux OS
211582 New AS3 NetStream.receiveVideoFps() should be NetStream.receiveVideoFPS()
211558 Flash Player movies soft-freeze
211504 getURL loading with UTF8 Japanese text data crashes IE6 on WinXP J
211498 crash if flash unloads while in a JS function called by ExternalInterface
211493 Audio locks up and starts looping on Linux, requiring the page with the player to be shutdown
211491 AS3/2 keyDown event (Code Keys) not firing for NumPad on Linux
211485 Flash previewer hangs the browser
211403 Found an audio sync issue after a pause and seek
211367 Memory leak when subscribing live stream with buffer time 0 published by Flash Media Encoder 2 in MP3 audio format
211362 One more time: An A/V Sync issue
211346 extra lines appear when a radial gradient is on top of a radial gradient
211343 Video does not resume after pause and seek
211298 Accessibility Plugin reads text from dynamic text field after tf.text has been set to ""
211264 Fms seeking while paused won't always generate an image
211263 FMS Video streams when seeking can go out of A/V sync
211186 H264 files : Loss of audio sync during the playback after seek
211171 site no longer automatically advances to next song
211120 new file icons for F4V, F4P, and F4A for the standalone player to register
211070 Intermittent (8/10) Crash on site
211067 NetStream.Seek.InvalidTime NetStatus event coming in different order from other events, breaking legacy content
211051 File Upload broken on leopard. FileReference throws cancel event when a file is selected for upload in the file-browsing dialog
211017 Scrubbing streaming flv doesn't work (possible injection)
210983 Incorrect sprite size and position on several platforms
210962 Player crashes when finish streaming a mp4 movie.
210957 Bitmap.draw() producing erratic behavior when subsequently drawn in bitmapfill
210915 Blur filter behavior uses black on edges instead of nearest pixel
210909 Fullscreen not painting over screen
210907 perform execstack and strip commands on executables
210901 [MP4] Handbrake videos made with certain settings play but halt batch at end of playback
210871 Smart Buffering: Few issues when streaming on2 files from Flash Media Server 2.0.4.
210854 scrolling images show stuttered or jumpy or jerky movement on playback in Vista
210846 [MP4] Errors in the 'ilst' parser
210844 Audio stutters
210809 widget hangs the browser
210808 Player crashes by scrubbing the playhead when streaming a mp4 file under rtmps, rtmpt and rtmpe connections.
210780 All the strings for the linux context menu leak [Memory Leak]
210769 Repeated use of shared objects in an event handler causes slow script dialog and hosed Settings UI to appear
210767 [LNX RHEL4] Crashes interacting with site
210746 When a release swf loads a debug swf, flash player doesn't look for the debugger
210720 Loader acts strangely in FullScreen mode
210717 Keyboard non functioning with the Netscape
210716 full screen does not respond to mouse events or display context menu
210715 Selecting printer dropdown cause hang with plugin on linux
210679 Certain videos look bad when played on Mac
210667 Context menu in full screen does not display, intermittently hangs
210645 streaming checks may unnecessarily reject valid socket policy files
210642 -root option sends sometimes a gtk warning, and sometimes a segfault
210641 Null Objects when iterating through children
210636 non-default locations not rejected for missing Content-Type
210593 Cannot open Settings manager page after clicking advance settings.
210574 Full episode videos don't stay in FullScreen Mode in Firefox
210573 Smart buffering: Artifacts with episodes
210562 dragging performs much poorer than 9r48
210561 Smart Buffering: NetStream.Play does not function after NetStream.Pause is called in this app.
210545 Netstream.receiveVideo parameter change is breaking existing content
210538 [MP4] Pauses due to buffer running out cause loss of sound sync
210532 Animation is skipped, causing audio and swf playback off sync.
210501 Enhanced Seeking driving audio/video out of sync on lower bandwidth like DSL
210485 Pause/Unpause on MP3 Files doesn't play from the paused location when setBufferTime is used.
210476 extraneous reload of content after close of sub-pane
210451 Player Crashing while playing a video through a serverside stream
210434 Player crashes when streaming this bad flv under an AS3 streaming application.
210433 massive performance degradation after being in fullscreen for a long time
210380 Refreshing the browser when AS3 content is loaded will cause the browser to crash.
210335 Flash content does not display
210333 Can't input local words with IME on linux
210331 [MP4] Lots of screen artifacts on certain trailers. PPC Only
210292 fscommand ("fullscreen", "false") crashes the Projector on Linux
210245 Can't input any words with IME
210209 [MP4]: When unpausing after seek while paused, video is black for one frame
210170 [MP4] AVC Profile 110 videos all appear purple on MacIntel machines
210097 Multiple SWFs hang page
210039 Leaking file descriptors to Flash resources
209833 Intermittent Flash Player 9.x crashes in Internet Explorer 6 and 7
209513 Changing IME conversion mode does not work on Mac X.5
208369 Consecutive shared object flush calls causes slow script error when used repeatedly in non-mouse/keyboard events
208212 Crash when loading external swf file in AS3
208125 page with 9 Flash video instances generates "R6025 - pure virtual function call" and crashes IE
208039 Flash plugin logs keyboard in Safari on Mac OS X
207349 Linux stalls on random test cases
206369 ByteArray.readMultiByte returns garbled characters from a binary file containing Win-1252 encoded characters.
147787 removeMovieClip should not fail for objects > kDisplayClonedEnd in depth

New File Extensions and MIME Types

2007-10-31T09:23:00.000-07:00

There have been a few questions around file types and mime types which should be used for the new video container format. To summarize this post, we will promote new file extensions and will stick with standard MIME types.

The new file extensions and MIME types will be the following:

File Extension	FTYP	MIME Type	Description
.f4v	'F4V '	video/mp4	Video for Adobe Flash Player
.f4p	'F4P '	video/mp4	Protected Media for Adobe Flash Player
.f4a	'F4A '	audio/mp4	Audio for Adobe Flash Player
.f4b	'F4B '	audio/mp4	Audio Book for Adobe Flash Player

These are pretty much in sync with what Apple does offer for their downloads. Why new file extensions? It will be an easy way to distinguish files which can be played back by the Flash Player. There are simply too many instances where .mov and .mp4 files can not be played back, or vise versa, a file compatible with the Flash Player might not play back in QuickTime, an iPod or other video device. I will also be working on a technical document to exactly outline what the Flash Player does support and what it does not support although my previous posts already pointed out that we are very close to full support of the H.264 standard with the exception of Extended profile and FRExt.

Why didn't we stick with .FLV? Technical reasons, mostly. Various technologies, including our own products expect that .FLV files have a certain file structure. In case of Flash CS3 a file which was renamed from .MP4 to .FLV would stall the application at import time. Not a good experience. In case of FMS file type plug ins can only handle unique file extensions. In this context it made it impossible to have files with the new container format using the .FLV file extension. We expect other tools to have similar issues.

It's not yet clear when our video tools will start using these extensions, although I expect it to happen sooner than later. The first one to be used will clearly be .f4v, the other ones are in 'reserved' state for now and not destined for a particular product right now. Why do I talk about this now in this case? Well, as we ready the Flash Player for release we did add new icons and file extension registration of these file types into the standalone Flash Player. So it's better to make this announcement now as you will discover this on your own very soon anyway.

If you ask what the FTYP column is, it is the ftyp atom within ISO 14496-12 files. If you have a custom encoder and are targeting the Flash Player you should add one of the ftyp major brand 4CCs mentioned above. This will make it much easier for servers to handle and recognize these files. And to prevent you from panicking now: the Flash Player will not even look at this atom. This atom is nothing more than a hint for tools handling ISO14496-12 files. A list of known ftyp major brand 4CCs can be found here. We'll hopefully make this list also at some point.

It might be a good time to update your IIS mime-type entries to include the new file extensions, otherwise IIS will refuse to play serve up these files. This is the same process as for flv files described in Technote 19439, but using the file extensions and mime types mentioned above.

Update, since I am apparently not clear enough and have to repeat myself: the Flash Player will not look at the file extension when loading files. It just means that if you are targeting the Flash Player or AIR we suggest to use these file extensions.

The obligatory post on Hydra

2007-10-04T21:26:00.000-07:00

Now that MAX 2007 in Chicago is over and the Flash Player team is somewhat settling for the final landing of Flash Player 9 Update 3 I've got to talk a little about Adobe Image Foundation, code named Hydra.

I have not been involved in this project as much as I would have liked to, with Bob Archer and Kevin Goldsmith driving most of the specifications and communication so far, nevertheless some technical questions in the context of Flash have been asked I should be able to answer. But let me make this post a little more approachable and explain the basics.

What is Hydra? If I would have to explain it in one sentence I would say: It is a shader language specifically tuned for 2D pixel graphics.

What is a shader language? Most of you will not know the answer to this. Let me try to put it into terms you might understand, with the assumption that you already have played with BitmapData objects in ActionScript 2.

Scenario: You have a bitmap and you want to exchange the red channel with the green channel. Easy enough, in Flash 8 or newer you can actually use the ColorMatrixFilter to do this. Lets say you do not have that filter, how would you do it? Well, this is what I would write in ActionScript 2:

import flash.display.*;

// create a green bitmap
var bitmap:BitmapData = new BitmapData(550,440,false,0x00FF00); 

// go through the whole bitmap pixel by pixel
for(var y:Number=0; y<bitmap.height; y++) {
    for(var x:Number=0; x<bitmap.width; x++) {

        // get a single pixel from our bitmap
        var pixel:Number = bitmap.getPixel(x, y);

        var red:Number   = (pixel>>16)&0xFF; // extract red
        var green:Number = (pixel>> 8)&0xFF; // extract green
        var blue:Number  = (pixel    )&0xFF; // extract blue

        // create a new pixel with the red and green channels flipped
        var newpixel:Number = (green<<16)|
                              (red  << 8)|
                              (blue     );

        // replace the ol' one with the new one
        bitmap.setPixel(x, y, newpixel);
    }
}
// now the bitmap is pure red

// show it
this.createEmptyMovieClip("bitmap_mc",this.getNextHighestDepth());
bitmap_mc.attachBitmap(bitmap, 2, "auto", true);

That sure looks ugly and slow. Now lets express the same as above using Hydra, and believe me, it will do exactly the same as the above AS2 code:


kernel FlipRedGreen
{
    void evaluatePixel(in image3 source, out pixel3 result)
    {
        result.grb = sample(source, outCoord());
    }
}

There are several things you will notice:

There are no for loops to go over the pixels. That's right, shader languages only express what it inside the loop and assume that this is the operation you want to perform on every pixel.

There are a bunch of strange types here! Not really: image3 stands for a bitmap with RGB data and pixel3 stands for an individual RGB pixel. If you would want to include alpha you could say image4 and pixel4.

What is sample()? It is the same as BitmapData.getPixel() and returns a pixel, in this case a pixel3 type.

What is outCoord()? It is the same as the x and y positions in the AS2 code but it returns a single value containing x and y without the need to specify them seperately.

How does the actual flip work? Now that is some of the beauty which comes with using a shader language. You can perform all kinds of operations directly with the component identifiers.

In this case we say 'result.grb =' which will flip the red and green channel. I could write 'result =' or 'result.rgb =' to just copy the pixel without a change. Or I could write any other combination like .bgr, .gbr to flip the different components.

If you really want to get fancy you can also write something like this:
'result = sample(source, outCoord()).rrr;' or
'result = sample(source, outCoord()).bbr;'.
In this case we make the component choice on the other side of the assigment.

In essence, no more bit fiddling anymore, you have a high level access to the actual component data.

This is one of the most simple examples I can think of, Hydra is much more powerful and expressive than that. So I invite you to take a look and get comfortable with it, this is something which will stay with us for a long time to come. And possibly not only for graphics, but let me talk about that in a future post. :-) I also promise that I will have another post specifically talking about the technical aspects of the Flash Player implementation. I can think of those and more:

How will I be able to access this in my ActionScript code?

How much faster will it really be than ActionScript code?

Where will I be able to use Hydra? Just filters and blend modes, or more?

Will Flash Authoring support Hydra?

Will the Flash Player take advantage of hardware acceleration?

Will this work in the Flash Player even if there is no hardware acceleration available?

I know you are clamoring for answers, though for most questions we have not figured them out ourselves just yet.

There is obviously one more question: "Instead of working on new features, when do you get your act together and ship a 64-bit version of the player?" :-) I always have a good laugh when we get this one, especially since it has been answered many times already.

Getting ready for showtime

2007-10-01T00:31:00.000-07:00

MAX is now underway and to celebrate this we just released another prerelease of Flash Player 9 Update 3 code named 'Moviestar', the version is 9.0.64.0.There have been numerous changes and fixes since the last beta, especially in the area of MP4, H.264 and AAC support. I figured I should summarize the bugs we fixed. In this area only as I was not able to keep track of the other stuff. As a warning, this is highly technical stuff, but I know that some wanted this information.

A frame ordering issue when using more than 1 reference frame has been fixed. There should be no limit anymore on how many reference frames can be used.

One multi core systems H.264 streams using the loop filter showed artifacts along the slice boundaries. This has been fixed by switching from sliced based multi threaded decoding to picture based multi threaded decoding.

Overlaying flash content would sometimes cause black artifacts during a seek because the underlying YUV buffer was invalid.

AAC streams would sometimes not play back if it contained SBR information. This usually happened when switching between SBR and non-SBR streams.

The H.264 codec would crash on octo (8) core machines.

AAC streams with sample rates higher than 44.1Khz could crash the Flash Player.

Corrupt or fuzzed AAC streams could crash the Flash Player.

The H.264 codec now supports FMO decoding, PicAFF and MBAFF decoding should be improved.

H.264 decoding should generally be about 10-15% faster because of the removal of a full frame copy. Note that this is only the case if a square pixel ratio is used, the video stream is not interlaced or cropped on the top, left or right.

The PPC version of the H.264 codec would call kill() during certain situations which resulted in a user experience similar to a crash. The reason behind this was the incorrect behavior of assert() in release builds under CodeWarrior.

The H.264 codec now has AltiVec versions for all the functions which are accelerated by MMX and SSE2 SIMD instructions on x86. Note that due to the extremely low available memory bandwidth of G4 and G5 PowerPC processors the overall performance will never match x86 CPUs. Refer to our system requirements page which should be up once the final version of MovieStar is released.

The mp4 parser will now parse variable chunk sizes correctly. Usually this bug manifested itself in bad or accelerated AAC audio playback.

The mp4 parser should now be much more robust when encountering corrupt or fuzzed files. Two new NetStream events have been added to capture invalid or unsupported files: NetStream.Play.FileStructureInvalid and NetStream.Play.NoSupportedTrackFound.

The H.264 codec could crash when trying to decode unsupported profiles like Extended, High 4:2:2 and High 4:4:4.

Past onImageData and onTextData events are now resent when seeking. This f.ex. allows previewing of cover art when scrubbing pod casts and will assure you will get the current timed text for any particular time. Note that also means that you will receive a lot of these events during a seek.

supported AAC aot types are now correctly parsed and validated, meaning we allow aot types 1, 2 and 5.

Some encoders (like Handbrake) embed aspect ratio information into the matrix of the track header box atom, which is inconsistent with ISO 14496-12. Despite this the Flash Player will now apply this matrix on the reported width and height in the onMetaData object. This does not affect the actual video stream size which can be different from what is reported in onMetaData and in this case usually points to a bug in the encoder.

Artifacts on the right and the bottom when mixing cropping and aspect ratios in H.264 streams should be gone.

Along with times, the seek points array will now also contain a byte offset which allows to determine if a particular seek point is reachable during a progressive download.

What is still outstanding before the final release?

There is an injection concerning scrubbing H.264 files. Currently no images are shown during scrubbing. You have to resume playback before the video is updated. There is a one line fix I have which did not make it in time for the 9.0.64.0 build.

When playing back video using FMS you can end up in a situation where you get 1 black frame when seeking. Another minimal low risk change will address this for the final build.

AAC streams using SBR do not play back in certain situations if the original sample rate is higher than 24Khz.

What are the known issues we will not address?

We still do not support Extended, High 4:2:2 or High 4:4:4 profiles. Unfortunately the most commonly non-professional codec out there which is x264 does not adhere to standard terminologies when specifying encoding options. This means you will likely run into a case were you encode using options which are part of the profiles we do not support. The most common option you will want to use is 4:4:4 support to obtain screen casts without YUV artifacts. This is not a bug, although I am sure we will continue to get reports on that. I strongly suggest you stick with Adobe's AME encoder which uses standardized terminologies defined in the H.264 specifications.

On the more controversial side we decided to postpone support for H.264 color profiles as the risk of adding tons of new SIMD code was too high at this point in time. Like MPEG-2 default H.264 streams use the color profile standardized under ITU-R BT.709 and in addition allows the selection of 6 more less frequently used color profiles. The streams will be displayed under the assumption that the content is ITU-R BT.601 meaning it will be a little brighter and slightly color shifted if the encoder did use ITU-R BT.709. Video buffs will frown upon this, but most end users will probably not notice it. We'll try to address this in one of the next versions of the Flash Players. I realize that it is important. For designers this also means that color matching with other Flash content will be more tricky for now.

A more sophisticated way of displaying field coded streams which we had prototyped but were not able to fully test will not make it into the final release. So field coded streams will simply be frame blended.

Corrupt AAC or H.264 streams or audio dropouts caused by the underlying OS can cause bad audio/video sync. This is due to an architectural limitation of the Flash Player which we will try to remedy for the next major version of the Flash Player, code named Astro.

What just happened to video on the web?

2007-08-20T21:09:00.000-07:00

That's a question you should ask with the announcement we made tonight. I think a lot will change. This is probably one of my longest and information packed posts ever, but I think it is important we put down all cards on the table. Lets summarize what new functionality Flash Player 9 Update 3 Beta 2 contains (for the impatient: It will be available on labs.adobe.com this afternoon):

An file format parser implementing parts of ISO 14496-12. In terms you might understand this means a very limited sub set of MPEG-4, 3GP and QuickTime movie support.

Support for the 3GPP timed text specification 3GPP TS 26.245. Essentially this is a standardized subtitle format within 3GP files.

Partial parsing support for the 'ilst' atom which is the ID3 equivalent iTunes uses to store meta data. This really more a de-facto standard which came through the ubiquity of iTunes, there is no official documentation on the format. Look here for an incomplete list of supported tags iTunes does use.

A software based H.264 codec with the ability to decode Base, Mainline and High profiles. This is also an ISO standard with the identifier being ISO 14496-10.

An AAC decoder supporting AAC Main, AAC LC and SBR (also known as HE-AAC). The corresponding ISO specification is ISO 14496-3.

That's pretty much what we say publicly. Truth is that these specifications are so complex that no one supports 100% of it. I realize that it will be important for Adobe to communicate exactly what is and what is not supported. We are working on this and will be trying to help novices and experts alike. For those who scream murder and accuse us of going with incomplete standards support let me tell you that ISO 14496-12 specifically allows for the definition of sub sets. 3GP is one of those. We did not extend or add proprietary extensions whatsoever to the mentioned standards above, it is a pure sub set.

Why now? Short answer: Because you wanted it. Long answer: We've been working on this for a while and this was planned to be part of the next major revision of the Flash Player. What was unexpected was how impatient a lot of our customers are :-) It seems many are trying to make choices when it comes to video technologies right now. We wanted to make sure that we would offer the best possible choices to them and set a signal that we are willing to embrace industry standards. No one believed that we would make this happen.

Unfortunately, and we realized while working on this: along with adopting industry standards also comes completely new terminology which seems designed to confuse non-insiders. This makes it difficult to pin down exactly what it is what we did and how you might benefit from it. It took me several months to just understand the basics in the ISO specifications. By now I might have lost the ability to boil it down into simple terms everyone can understand. But I'll try anyway. :-)

Lets talk about actual functionality you can leverage in the Flash Player. Now I am getting really technical:

You can load and play .mp4,.m4v,.m4a,.mov and .3gp files using the same NetStream API you use to load FLV files now. We did not add any sort of new API in the Flash Player. All your existing video playback front ends will work as they are. As long as they do not look at the file extension that is, though renaming the files to use the .flv file extension might help your component. The Flash Player itself does not care about file extensions, you can feed it .txt files for all it matters. The Flash Player always looks inside the file to determine what type of file it is.

A new version of FMS is upcoming and will support the new file format. This is powerful stuff. Simply drop video files you might have encoded using one of the countless tools out there onto the server and it'll stream. Even if the moov atom is at the end of the file. Ah, that is something I have to mention as you are 100% likely to fall into this trap:

If you use progressive download instead of FMS make sure that the moov atom (which is the index information in MPEG-4 files) is at the beginning of the file. Otherwise you have to wait until the file is completely downloaded before it is played back. You can use tools like qt-faststart.c written by our own Mike Melanson to fix your files so that the index is at the beginning of the file. Unfortunately our tools (Premiere and AfterEffects etc.) currently place the index at the end of the file so this tool might become essential for you, at least for now. We are working hard to fix this in our video tools. There is nothing we can do in the Flash Player and iTunes/QuickTime does behave the same way.

The Flash Player will display the first supported video and audio track it finds in a file. Subsequent audio and video tracks are ignored and not selectable right now. This covers the majority of files out there on the web, only in rare instances do you have additional audio tracks f.ex. But I believe that for the web you would rather create several versions of a file anyway to save bandwidth. One of next major revisions of the Flash Player will add new APIs to enhance this most likely. Our goal was not to add any new APIs for this release.

Video needs to be in H.264 format only. MPEG-4 Part 2 (Xvid, DivX etc.) video is not supported, H.263 video is not supported, Sorenson Video is not supported. Keep in mind that a lot of pod casts are still using MPEG-4 Part 2. So do not be surprised if you do not see any video. We should be close to 100% compliant to the H.264 standard, all Base, Main, High and High 10 bit streams should play. Extended, High 4:2:2 and High 4:4:4 profiles are not officially supported at this time. They might or might not work depending on what features are used. We have no artificial lower limit on B-frames or any problems with B-pyramids like other players do. We also decode field coded streams, although this beta displays the images progressively using the weave method. The final release will be blending the two fields. There are still a couple of bugs with frame ordering/timing I need to fix in the Flash Player itself for the final release. And there is also a problem with files using the loop filter on dual core machines which causes horizontal artifacts along slice boundaries, which is my bad. The fix for this did not make it into this beta. Overall though and leaving out the bugs I listed here which are my fault, the H.264 decoder is a remarkable piece of engineering, it is provided to us by MainConcept. It weights in at less than 100KB of compressed code which is quite an achievement for such a complicated standard.

Audio can be either AAC Main, AAC LC or SBR, corresponding to audio object types 0, 1 and 2. We also support the '.mp3' sample type meaning tracks with mp3 audio. MP3inMP4 which intends to do multi-channel mp3 playback within mp4 files is not supported. Also, the old QuickTime specific style of embedding AAC and MP3 data is not supported. It is unlikely though that you will run into these kind of files.

3gp timed text tracks. Any number of text tracks are supported and all the information, including esoteric stuff like karaoke meta data is dumped in 'onMetaData' and a new 'onTextData' NetStream callback. Language information in the individual tracks is also reported. That means you can have sub titles in several languages. Study the 3GPP TS 26.245 specification to see what information is available. Note that you have to take care of the formatting and placement of the text yourself, the Flash Player will do nothing here. Time for you to start working on one of those components which do that. You can use MP4Box to inject text data into existing files.

Meta data stored in the 'ilst' atom. This is usually present in iTunes files. It contains ID3 like information and is reported in the onMetaData callback as key/value pairs in a mixed array with the name 'tags'. ID3V2 is not supported right now. An incomplete list and link to tools which can edit these tags is available here.

Since these files contain an index unlike old FLV files, we can provide a list of save seek points, e.g. times you can seek to without having the play head jump around. You'll get this information through the onMetaData callback in an array with the name 'seekpoints'. On the downside, some files are missing this information which also means that these files are not seekable at all! This is very different from the traditional FLV file format which is rather based on the notion of key frames to determine the seek points.

Unencrypted audio book files contain chapter information. We expose this in the onMetaData callback as an array of objects with name 'chapters'.

Image tracks encoded in JPEG, GIF and PNG are accessible. Unfortunately only in AS3 as I pass this information as a byte arrays through a new callback 'onImageData'. You can simply take that byte array and use the Loader class to display the images. Most often these images represent cover artwork for audio files. TIFF image tracks are not supported, you might come across files using this. Also note that we support the 'covr' meta data stored in iTunes files, these are also accessible as byte arrays.

Will it be possible to place H.264 streams into the traditional FLV file structure? It will, but we strongly encourage everyone to embrace the new standard file format. There are functional limits with the FLV structure when streaming H.264 which we could not overcome without a redesign of the file format. This is one reason we are moving away from the traditional FLV file structure. Specifically dealing with sequence headers and enders is tricky with FLV streams.

Will it be possible to place AAC streams into an FLV file structure? Yes, though the same limitations as for H.264 apply.

Will the Flash Player play back multi channel AAC files? It will play them, though the sound is mixed down to two channels and resampled to 44.1Khz. We are targeting multi channel playback for one of the next major revisions of the Flash Player. This requires complete redesign of the sound engine in the Flash Player which dates from circa 1996 and has not been improved since.

Will the Flash Player be limited to 11Khz, 22Khz and 44.1Khz sampling rates like for MP3? No, we support all sampling rates from 8Khz to 96Khz. I implemented a 32 tap Kaiser Bessel based FIR filter which resamples the sound to 44.1Khz, retaining high quality. The most common sample rate combinations have a hard coded number of phases. In case of a 48000 to 44100 Hz conversion the filter has 147 phases f.ex. Even better: Flash Player Update 3 Beta 2 now can play back any MP3 sampling rate leveraging the same code I implemented for AAC. No more chipmunks. Ever. Err, this is actually kind of major as I have seen complaints about this bug for years :-) I fixed this problem in the AS3 Sound class, though it was using very low quality resampling. This change I made this time will fix it even for AS2 and sound in FLV files while retaining excellent quality.

Will it be possible to place On2 VP6 streams into the new file format? Not right now, we are still trying to figure out if it is possible for us to support this.

Can you play files protected by FairPlay? No.

Do we support MPEG-4 BIFS or other esoteric stuff (scripting, VRML etc.) from the MPEG-4 Systems specification? No. Whatever is not listed above we do not support.

Do we support SMIL? No. You can easily write your own SMIL parser in ActionScript though.

Can you use the Sound class to play back AAC/.mp4a files? No, you have to use the NetStream class. We are now getting into a situation where there is not much difference between audio and video files anymore. They are the same essentially. Hence we figured we should not further add confusion and allow to do things ten different ways which would also increase the Flash Player binary size. My guess is that we will enhance the Sound class in the future but it might go into a different direction and will not be dedicated to pure playback of static files anymore.

Here is a list of data which is reported in onMetaData:

duration - Obvious. But unlike for FLV files this field will always be present.

videocodecid - For H.264 we report 'avc1'.

audiocodecid - For AAC we report 'mp4a', for MP3 we report '.mp3'.

avcprofile - 66, 77, 88, 100, 110, 122 or 144 which corresponds to the H.264 profiles.

avclevel - A number between 10 and 51. Consult this list to find out more.

aottype - Either 0, 1 or 2. This corresponds to AAC Main, AAC LC and SBR audio types.

moovposition - The offset in bytes of the moov atom in a file.

trackinfo - An array of objects containing various infomation about all the tracks in a file.

chapters - As mentioned above information about chapters in audiobooks.

seekpoints - As mentioned above times you can directly feed into NetStream.seek();

videoframerate - The frame rate of the video if a monotone frame rate is used. Most videos will have a monotone frame rate.

audiosamplerate - The original sampling rate of the audio track.

audiochannels - The original number of channels of the audio track.

tags - As mentioned above ID3 like tag information.

Here are some good links to get an understanding of what MPEG-4, H.264 and AAC are:

http://forum.doom9.org/showthread.php?s&amp;threadid=62723
http://forum.doom9.org/showthread.php?t=96059
http://en.wikipedia.org/wiki/H264
http://en.wikipedia.org/wiki/Advanced_Audio_Coding
http://daringfireball.net/2007/04/some_facts_about_aac

Let's put together some thought up scenarios I would imagine are important:

You created a pod cast for iTunes and happily distribute over this channel. Now you want to add value to it and easily make it accessible over the web without special plug-ins, reaching an audience which does not have QuickTime installed. Well, this new feature will allow you to do this. You can take your existing podcast in .m4a format and present it on any web page through the Flash Player. Add more value by adding interactivity and branding if you want to. The possibilities are endless.

Your media company has made or is about to make a significant investment into web video or video archiving. You are wondering what format you should choose. Video for Flash reaches everyone now, but the format is not an 'industry standard' so you have the fear that content you will create will become obsolete and unsupported at some point. Flash Player 9 Update 3 comes to the rescue: MPEG-4 is an extremely well documented ISO standard and completely vendor independent. And by using the Flash Player now you get instant gratification for viewers.

You want to get best the possible quality out of your video and do not want to be tied to a particular encoding solution. You also like open source software to do all of the work you need to do to encode video. A combination of libfaad, x264 and MP4Box which are all licensed under the GPL will do exactly that, albeit with little usability and requiring lots of expertise. But it will now play just fine through the most distributed run time in the world, the Adobe Flash Player.

Those are immediate benefits, there are plenty more when we look ahead. Let me mention a few of them:

H.264 will be supported natively by most new graphics cards. NVidia, ATI and Intel have made a commitments to have full support for it. This means better than HD video on your PC will become possible in the not so distant future.

There are hardware based H.264 encoders which encode at better than real time. This is important if you need to be quick to market like f.ex news organisations.

Digital TV, especially in Europe is quickly adopting H.264. The interoperability with the web will open new doors for a lot of media companies.

AAC SBR offers demonstrable advantages over plain MP3, think 5.1 channel surround sound f.ex. While the Flash Player does only support 2 channels output at this time, there is opportunity to go beyond that.

And last but not least here are some things I will not give a complete answer to since they are begging for controversy:

Comparing H.264 against other video codecs, might it be performance or quality. I've looked at the comparisons out there, they are at best subjective, most of the times outright marketing bull and almost always completely biased. My take is: Take a good and well accepted encoder and compare the results yourself. Your mileage will vary. And that is fine. Quality is not the main reason Flash Player 9 Update 3 has H.264 support.

Tell you if On2 VP6 is better or worse than H.264. Truth is that they have different strengths, not only performance and quality wise. It totally depends on your individual situation of what fits best. The Adobe Flash Player now offers more choice which is more important than anything else.

I am not in a position able to explain to you why we will not allow 3rd party streaming servers to stream H.264 video or AAC audio into the Flash Player. What I can tell you is that we do not allow this without proper licensing. Refer to Adobe's friendly Flash Media Server sales staff for more information.

I can also not help you with anything regarding broadcast fees for commerical use of H.264 and AAC streams. Please refer to the FAQ Adobe provides which usually point to contacts at MPEG-LA and Via Licensing. A summary of licencing terms for H.264 is available here.

Multi core support

2007-06-12T10:57:00.000-07:00

As I mentioned in Flash Player Update 3 we finally realized that multi-core CPUs are here to stay. So why not follow the times and take advantage of it? As most of you hard core Flash developers know, rendering is a huge bottleneck. I've seen a couple of blog post complaining that their second core/CPU is not doing anything when they run the Flash Player. Well people, this is about to change in this update.

Flash Player Update 3 takes advantage of multiple CPUs in many different areas. The first one is obvious: The vector rasterizer. The Flash Player will split the rasterization workload by dividing the stage into horizontal stripes. In case you have to 2 cores/CPUs, the top half is rendered by CPU 1, the bottom half by CPU 2.

How much performance improvements will you see? That depends, unlike with a 3D ray tracer the workload is not totally independent and there is some additional overhead to handle multiple threads. As a rule of thumb you could say that you get an additional 25-33% performance improvement per CPU. That does not sound like a lot, but keep in mind that the Flash Player still does some stuff which can not be multi-threaded.

The second area we focused on a are bitmap filters. If you deal with large movie clips and use lots of filters on them you will see a nice performance jump. Again the workload is split into horizontal stripes. The following filters are supported:

DropShadowFilter

GlowFilter

GradientGlowFilter

BevelFilter

GradientBevelFilter

BlurFilter

ConvolutionFilter

ColorMatrixFilter

DisplacementMapFilter

Third, compositing and color transformations of bitmap cached object are also accelerated. The BitmapData APIs are NOT multi-threaded yet, hopefully we get around doing this at some point.

Fourth, VP6 video decoding now happens in a separate thread, independent of rendering and post processing of the video. In most cases that means a dramatic improvement in performance, especially if you did reach a limit performance wise with your current content despite the fact that you had a dual core machine. On my Athlon X2 4800 I can now play a true 1920x1080 pixels video at 30 frames/sec without a single dropped frame. Really cool if you ask me. It was a reason for me to buy a new toy: a Canon HV20 which can do 1080p at 24 frames/sec (actual pixel resolution is 1440x1080). Perfect for the web. Now if these video files would not clog up my drive, they are really huge...

Why is the Sorenson Spark codec not multi-threaded? Because of its design: It does not have a dual YUV buffer setup which allows us to separate blitting and video decoding.

Since I am talking about VP6 already let me mention some of the more generic improvements in VP6:

I added a new gaussian noise filter which is turned on when post processing is active. It lessens the effect of blockiness and makes videos more look like movies. If you are turned off by this change, please let me know as soon as possible. In most of the cases it is seen as a improvement, but you designers should tell us if things still look okay.

We fixed a long standing issue with VP6's automatic post processing selection on Athlon X2 class CPUs. It never worked correctly on these machines. This is due to a broken CPU driver on stock installs of Windows which affects the QueryPerformanceCounter Win32 call. Incidentally the new version of QuickTime and the Real Media Player seem to have problems related to this API, playback stutters heavily on my machine with some content because of it. I usually fix this problem by setting the process affinity to a single CPU in the Task Manager. That's what you get for not testing on AMD CPUs :-)

Entropy decoding and some parts of the decoder should be faster, you should see a generic 10% improvement in CPU usage.

Mip map what?

2007-06-11T10:00:00.000-07:00

True to my reputation here at Adobe I added a miniature feature into Flash Player 9 Update 3 (9.0.60.120). It is Mip mapping. The Wikipedia entry is nice, but there is an in depth article I highly recommend to anyone interested in the subject: Part 1 and Part 2. As described in the article we use the fast and simple box filtering as apposed to the more complex methods which are not really suitable for real-time rendering. So no Kaiser in the Flash Player. Neither do we have trilinear or anisotropic filtering, but then again they make more sense with real 3D graphics. And since we are still stuck in software only land for bitmap rendering I was not interested in a performance loss either.

Why do I call it a miniature feature? Because the whole feature occupies less than 1Kb of compressed code in the binary. This is the kind of changes we engineers in the Flash Player love. Extremely small in code size, large in impact.

If you feel kind of overwhelmed with the descriptions in these articles let me summarize that the feature does in more simple words: This feature improves the quality and performance of downscaled bitmaps. Not slightly downscaled, but anything which is scaled down by more than 50%. Mip maps are nothing more than precomputed smaller and higher quality versions of an original bitmap. They are used instead of the original bitmap when something is downscaled a lot.

A few words about the Flash Player implementation and its limitation might be worth talking about here:

Mip maps are only created for 'static' bitmaps, e.g. anything like a JPEG, GIF or PNG you display via loadMovie(), a bitmap in the library or a BitmapData object. They do not apply to filtered objects or bitmap cached movie clips.

In case of video things are more tricky. If smoothing is turned on the mip mapping applies since this is overall faster than rendering the original bitmap. For non-smoothed video mip mapping is not used since it would make things much slower.

Mip maps level generation stops when it encounters an odd width or height. What does that mean? In the most extreme case it means that if you have a bitmap with an odd width or height pixel value to begin with, no mip map can be generated at all and therefore you will not see any benefit. This is unfortunately hard technical limitation. That also means that 'perfect' mip maps are generated from bitmaps which have a width and height which are 2^n, e.g. 256x256, 512x512, 1024x1024 etc. On average though it is enough to have bitmaps sizes which are dividable by 8, meaning you get at least 8 levels, f.ex. 640x128 -> 320x64 -> 160x32 -> 80x16 -> 40x8 -> 20x4 -> 10x2 -> 5x1.

The quality improvements are more visible when smoothing in turned on for a particular bitmap.

The most immediate effect of this feature can be seen with Papervision3D. The Rhino demo f.ex. shows much less aliasing in the textures when you rotate. Also, a couple of frames per second more are displayed with this demo.

Good and well, but how does 'normal' Flash content benefit? Any time you create something like an image gallery with small thumbnails which are based on larger bitmaps you will see better bitmap quality and slightly higher performance.

Will this feature ever make things look worse? Not unless you do something really dumb like downscaling heavily aliased images and still expect it to look pixelized.

I made up a quick and dirty demo which shows the reduced aliasing effect: http://www.kaourantin.net/swf/mipmap.html Make sure you have Flash Player Update 3 Beta 1 installed, otherwise the two samples will look exactly the same.

What about control on this feature? Can you select the threshold or select your own mip map bitmaps? No, right now everything is automatic. That might change in a next major revision of the Flash Player. And the threshold value, well, let me simply tell you that we use the standard OpenGL value which is <= 0.5. If you want to know how this actually works, here is I would express it in pseudo ActionScript 3, you graphic experts will get it right away:


var bitmapData:Array; // contains the mip map bitmaps
var m:Matrix; // the affine matrix to used for display

var i:int = 0;
while ( Math.sqrt(m.a*m.a+m.b*m.b) <= 0.5 &&
        (bitmapData[i].width % 1) == 0 &&
        (bitmapData[i].height % 1) == 0) {

   // switch to next mip map level
   i++;
   m.a /= 2.0;
   m.b /= 2.0;
   m.c /= 2.0;
   m.d /= 2.0;
   m.tx /= 2.0;
   m.ty /= 2.0;

}
mc.draw(bitmapData[i],m);

At last I should mention that mip maps apply to any SWF version 9 or newer content. SWF version 8 or lower will not use mip maps since we feared that this would impact on backwards compatibility.

Flash Player Update 3 Beta 1

2007-06-10T21:04:00.000-07:00

Flash Player Update 3 Beta 1 (build 9.0.60.120) is now ready for download here. This is probably the first time we have done a dot release as big as this one. We've been extremely busy over the past few months since Flash Player Update 2 (build 9.0.r45), so now it is finally time to talk about the improvements we have made. There are tons of them, so I'll use this post to summarize those I know something about and I have been working on. Stay tuned for more detailed posts of mine explaining the technical details behind these and how you will be able to leverage them:

Mip map support for all bitmaps for Flash 9 or newer content. This improves the quality and rendering performance of downscaled bitmaps. Perfect for thumbnails and such. Even better, Papervision 3D content now automatically looks better and should be slightly faster when large textures are used.
Multi-threaded vector renderer. Now we take advantage of up to 4 Cores/CPUs for vector rendering.
Multi-threaded bitmap filters. Same as above but this applies to bitmap functionality specifically instead of the core vector rasterizer only.
Multi-threaded video decoding. The VP6 video codec will now run in a separate thread if a multi-core system is detected which leaves the main thread to do rendering and post processing of the video. With this true 1080p video is now possible on most modern dual core machines. Also, the responsiveness is improved with this change. The Sorenson codec on the other hand did not get this change for technical reasons.
Full-screen mode with hardware scaling. Probably the biggest new feature in the Flash Player Update 3. This leverages DirectX on Windows and OpenGL on OSX. There is an new API to control the behavior which was required since we could not change current behavior and we wanted to give the maximum flexibility possible. I know you are probably eager to use this feature, we will post more information on this on labs.adobe.com soon (Update: Link to labs page is active). I'll also will give you much more technical details in an upcoming blog post.
Less tearing in the new full screen mode. We now have some code which will try to do VBL syncing. It's still a work in progress but we hope we can fix the remaining issues.
Going into full screen mode has a zoom transition effect. The beta does not work perfectly right now, but we want to get feedback if this is acceptable to end users. We will not expose an API to access/control this, either we'll leave it in and fix the remaining bugs or it is out. Also you might notice that this even affects the current full screen mode, something we will remove in the final release.
The Linux plugin now uses the XEmbed protocol. This is work in progress. The downside is that konqueror and Opera do not support this right now, so the Flash plugin will not work until these vendors update their plugin support. Also we are seeing decreased performance because GTK lacks somewhat in the the basic graphics API department. I'll explain in a later post.
Tons of performance tweaks and bug fixes. Looking the the current bug database statistic we fixed 371 bugs since 9.0.r45. Fixed really means fixed, it does not include duplicates, unreproducible bugs etc.
Much more.

A word of warning, this is a beta version! Do not use this version in a production environment. There are several known issues with the new features and while they might work on your machine they will not on others. We are obviously interested in the others and are looking for any issues you might encounter.

The limits of software rendering

2007-02-01T16:46:00.000-08:00

I have been following the progress of Papervision3D recently, the 3D engine for Flash. Here at Adobe we are always interested in applications which challenge the Flash Player performance wise. We spent some time to profile some demonstrations to see if there is anything we can do performance wise. The short answer is: not really.

What is the problem? Well, there is a limit what software can do on current CPUs. The limit can be squarely attributed to memory bandwidth and the size of L1 and L2 caches. I thought I'd explain the point by showing some actual profiling data I collected with the demonstration mentioned above. You can reproduce the same results including seeing the actual assembly instructions with any 3rd party profiler tool out there.

For testing I used a pretty beefy system, an AMD Athlon 64 X2 4800+. CodeAnalyst was my profiler of choice since this is what AMD provides for their CPUs, Intels VTune does the job just as well though if you have an Intel CPU.

Here are the top three loops which show up in the profiler and together take about 40% of overall CPU time:

The first loop does the affine bitmap blitting, meaning it handles rotation and scaling. This loop takes about 20-25% of overall time. Its characteristic is that memory access is almost always non sequential, meaning you get lots of L1 and L2 cache misses.

The second loop does a simple copy from one buffer to another. Some will recognize this code, this was adopted from the AMD Athlon Processor x86 Code Optimization Guide. This loop takes about 10-15% of overall time.

The third does a blur for a radius which is power of two, I have presented this code in a blog post before. This loop takes another 10-15% of overall time.

So what can you determine from the above? Look at the CPU clock column. This is the number of total cycles spent on an individual instruction. Any memory access, especially random access like the affine bitmap rendering is the most expensive operation. It really does not matter at that point, without GPU hardware support there is pretty much nothing we can do anymore on modern CPUs when it comes to graphics performance.

Obviously there are other areas we can improve, the other 60% of overall time have maybe another 10% overall performance bump in store if we fundamentally change some algorithms and fix bottlenecks in Tamarin. Although in this example ActionScript execution really does not account for much, it is really all related to graphics code. In the end the conclusion remains: It is memory access time which kills us.

What is even more surprising though for some is that even multi core systems will not help dramatically. In most cases the memory bus and even the L2 cache are shared among the cores.

58th Annual Emmy Awards Technology & Engineering

2007-01-18T14:29:00.000-08:00

On January 8th I had the privilege to attend the 58th Annual Emmy Awards for Technology & Engineering in Las Vegas and receive, together with other co-workers, an award. In case you did not know, Adobe Inc., among other well known companies in this industry got an award for advancing streaming technology, specifically video streaming over the internet.

The actual team at Adobe working on Flash Video technology has remained small over the years. So it was fairly easy to get the most important people from our engineering teams to be on stage to receive the award, something Microsoft and Apple were apparently not able to do:

(Engineers in suits! I am the second guy from the left, low resolution is on purpose to protect the innocent.)

And here is a close-up of the award:

Why did we get the award? Well, some will understand and some will not. I will dwell into my personal opinion at some point, right now I do not have the energy to defend myself in that regard :-) It is interesting though that all the major players did receive the award, which is more of a sign that it is really about the acknowledgement that video on web has finally become mainstream. This has been the dream of many for decades, but it really started taking off 2 or 3 years ago.

I am also happy for another award winner this year: John Carmack. He received two awards, one for Doom and one for Quake. These two games were definitely ground breaking and help shape the gaming industry to what it is now.

A non-beta Flash Player 9 for Linux release

2007-01-16T21:45:00.000-08:00

We just released another version of the Flash Player 9 for Linux, the version number is 9.0.31.0. What is different this time is that we do no call it a beta anymore and that it is available on our main download site (Click on the Linux x86 link). For the impatient ones, the direct download of the tarball is here, and the .rpm file is here:

Alright, so far so good... That does not mean that this is a release which I consider feature complete. We do however consider this version to be vastly better than any previous Flash Player for Linux. Since customers were starting to ask for a final version we decided to feature freeze what we have. So two months ago we moved the Flash Player 9 for Linux code into a separate code branch for stabilization. QE (Quality Engineering) has been hammering on this branch for all this time. Every bug fix had to be approved by management. The build you can download now (9.0.31.0) represents that branch.

The internal unstable branch is currently open for all developers and for the first time we build the Windows, Mac and Linux versions on a daily basis from the same code branch. There are various large work items which still need to be tackled and we are in full swing getting this ready as soon as possible. These changes are disruptive for the stability of the build and will be for a while.

Here is a list of some of the Linux specific items we are working on right now, it is far from complete and each of the items might or might not make it into the next version:

XEmbed support (specifically GtkPlug). We have tons of bugs which can only be fixed this way.

WMODE support. This will be tricky since we need to coordinate this feature with Mozilla as the browsers on Linux have no support for this right now. Also note that XEmbed support is a prerequisite for this feature to work at all.

Support for Opera. Hopefully we will get it into a state where it runs stable for more than 5 minutes. Opera 9.10 solved a lot of issues, but not all.

Rewrite the braindead/non-standard http stack in the standalone player. Right now this bug can make the standalone player timeout and block for a long time if you need to open any external URL (local files are fine). I'll spare you the details, but this code is really embarassing in my mind.

Try to fix other functional bugs which require larger infrastructural changes.

Finish the flashsupport add-on library. Adding camera, microphone support etc. Move the project to sourceforge.net or similar site. BTW, it's my fault that this has not seen too many changes lately since I own this piece. :-)

Add full screen support.

64bit work is ongoing. You can follow some of the daily progress in the Tamarin CVS tree. (Alright, not too much happened end of December. Forgive us, the holidays are always a little slow in a company like Adobe.)

So what changed between Flash Player 9.0.21.78 beta and 9.0.31.0? Here is a partial list of Linux specific bugs we fixed:


192642 Crash during "Browse multi"   
191932 After plug-in player installed, firefox cannot launch (crash while launching)  
191298 Firefox 1.5 crashes during tests execution (automated section) 
191514 Non Functioning rtmps not functioning 
192615 Crash hang navigating away from camera/mic  
192747 LocalConnection not functioning between standalone players 
192653 change standalone name from gflashplayer to flashplayer 
192399 Crash - GrowableBuffer and/or Generated Code 
191741 crash when "Allow" is clicked in camera settings UI popup when no camera driver is installed on SUSE 9 
192006 Clicking on URL failed to launch websites 
191749 mouseDown count advancing more than once per click - stand alone 
187060 Rotating the image moves image out of the panel 
177699 Bug in detection kit in combination with Linux player 
191427 rtmpt not functioning 
192036 Incorrectly Functioning File->Close is still selectable in the projector 
191843 website crashes with a segmentation fault.  
191267 Unable to copy/paste text from text fields using context menu. 
191857 remove all printfs from release builds of the plugin and standalone

(Update: I started this entry last week to start listing the fixed bugs, but actually posted the news January 16th. Darn. So some you you saw January 11th as the post date... I am really sorry for the confusion...)

Flash Player 9 for Linux Beta 2

2006-11-20T17:44:00.000-08:00

We have just refreshed the beta of Adobe Flash Player 9 for Linux, the current version is now 9.0.21.78. This version fixes several critical issues, especially some of the sound problems should be a thing of the past. So we encourage you to update.

Here is a partial list of bugs which were fixed (GNU/Linux specific, general bugs are mentioned in the labs release notes):

188992 - ALSA 2-second stopper problem
189976 - Leaking X11 connections
183202 - Flash crashes with X COMPOSITE
187715 - Asian characters are not displaying as expceted
187409 - Artifacts when initially playing + video stream freeze when selecting second video clip
189437 - Device text rendering performance is less than optimal
187850 - OS freeze right clicking flash movie after clicking another active link
184706 - browser crashes when trying to run use getters or setters from a different scope
188398 - in all the System.codePage 'FileReference', when user attempts to browse, a CRASH is generated
184549 - Mandriva 2006. Installing libflashplayer.so crashes browser
181584 - GUI not rendering correctly. Green out edges on inner frames.
189254 - wipe effect doesn't look right on components if they have no descenders
187885 - Some part of the image is shown out of panel
188349 - Clipping and Masking is not working correctly for Flex Containers when you scroll Text
187689 - Text can become garbled when window redraws.
188709 - Browser hangs
187490 - Crash accessing flash game
188805 - ALT key does not function in Linux player. (Ubuntu, others?)
188089 - RichTextEditor's formatting are not preserved during cut/copy/paste only in linux player
188997 - Selection copy won't paste into Flash.
186982 - activate, deactivate, mouseLeave events don't get fired when clicking inside/outside the player instance
187170 - caps lock and num lock are detected when flash player instance, or even the browser, does not have focus
187741 - change event not triggered when entering text in text field

If you are using flashsupport.c you should update it since I have removed the ICU dependency. I am currently revising this internally since I have received some great feedback on it. You should see these changes in the next version. Hopefully I'll also be able to finally check this into our public SVN repository at that time, since it really belongs there, not on a random web server, without a correct installation script or working versioning system.

Another update: Please note that this version does NOT work with Opera (nor did 9.0.21.55 btw). We know about it and are working with Opera to resolve it. Some sites might work, but most will eventually display a gray or pink rectangle, meaning that the Flash Player did crash. We know it is a problem with the way X11 is used (if you switch X11 to use synchronous mode it should work).

Also, we do NOT support GTK1 browsers and probably never will. That means any FireFox version older than 1.5 will probably not work and older Mozilla/SeaMonkey versions will also fail. The final version will have a safety check for this and the Flash Player will not try to run at all in this case. Right now you will most likely crash.

SpiderMonkey's relative Tamarin joins the family

2006-11-06T17:29:00.000-08:00

Today Adobe announced that the ECMAScript 4 compatible virtual machine in the Adobe Flash Player has been contributed to the Mozilla project under the name Tamarin. It is the single largest contribution to the Mozilla foundation since its inception and consist of about 135.000 lines of source code. The engine is fully open source using the standard Mozilla license, with the Mozilla foundation retaining full ownership.

What does this mean?

I anticipate that Tamarin will become the reference implementation for ECMAScript 4 VMs. Most of you have probably not realized yet, but ActionScript 3 IS an ECMAScript 4 compatible VM (or at least the closest one as one of my coworkers wants me to point out). Yes, it is the implementation of a standard and not something Adobe has made up. Yes, I know ECMAScript 4 was and still is preliminary, but the changes which will be adopted before it goes official will probably be extremely minor (depending on which further additions get approved though).

Adobe does NOT open source the Flash Player by doing this. Just the Verifier, JIT, core frameworks and the garbage collection engine are now open source. Also important to note: The java based compiler which converts ECMAScript 4 to the byte code understood by Tamarin is not included in this agreement. Instead the focus will be an ECMAScript 4 based compiler, which will be required for eval() to work anyway since we don't want to pollute Mozilla with too much Java code. Adobe Flash Player 9 does not support eval() to keep the binary size of the Flash Player to a minimum. Our focus on the Java compiler was required to support Flex Builder 2.

The plan is that Mozilla and the Adobe Flash Player will share the exact same code base now and in the future. There is no plan to fork the virtual machine at any point. Changes which go into the Mozilla source tree will be directly adopted by Adobe and vice versa.

Mozilla might adopt the conservative garbage collector even for SpiderMonkey. Again, what most people do not realize is that we made the switch from a simple reference counting system to a more advanced mechanism for ActionScript 1&2 while maintaining backwards compatibility. This was an incredible challenge. In case of the Adobe Flash Player this resulted in about half the memory requirements in many situations. This would be a huge boon for AJAX based applications, in addition to resolving circular reference issues if they still exist in SpiderMoney.

If you study the source code you'll realize that a 64bit port is NOT a recompile away. We are actively working on the 64bit port, the source code right now is still 32bit until the changes required are stabilized.

Flash/XComposite crasher in X11

2006-10-30T12:37:00.000-08:00

This one has been around for a while: When you enable the XComposite extension in X11 and run your desktop using a 15 or 16bit screen depth, the Flash Player will always crash with this error message:

The program 'firefox' received an X Window System error.
This probably reflects a bug in the program.
The error was 'BadMatch (invalid parameter attributes)'.
(Details: serial 115 error_code 8 request_code 143 minor_code 3)
(Note to programmers: normally, X errors are reported asynchronously;
that is, you will receive the error a while after causing it.
To debug your program, run it with the --sync command line
option to change this behavior. You can then get a meaningful
backtrace from your debugger if you break on the gdk_x_error() function.)
Segmentation fault (core dumped)

There are two workarounds for this:

Set the screen depth to 24bit. If you have enough VRAM available this is a good move anyway since you'll get much better performance and quality in general when using Flash. Other applications probably also benefit, there should be less color banding in general.

Set the XLIB_SKIP_ARGB_VISUALS environment variable to 1 in one of your init scripts before X11 is started.

So why have we not fixed this in the past? Well, I've not written the code in question and IMO it's even questionable if it is a bug in Flash, depending on what XMatchVisualInfo API is really supposed to do. The docs really do not provide too much information here. The core of the problem is this code in the Flash Player:

int depth = 0;
int mode = 0;
XVisualInfo vi;
static struct
{
  int depth, mode;
} formats[] = {
  { 24, TrueColor},
  { 32, TrueColor},
  { 16, TrueColor},
  { 15, TrueColor}, 
  { 8,  PseudoColor},
  { 0,  0}
};
for (int i=0;formats[i].depth; i++ )
{
  if(XMatchVisualInfo(display,    
    DefaultScreen(display),
    formats[i].depth, 
    formats[i].mode, 
    &vi)) {
      depth = formats[i].depth;
      mode = formats[i].mode;
      break;
  }
}

If XComposite is turned on and your screen depth is 16bit this loop will break when the depth is 32. That depth is used to create the XImage the Flash Player will use for its backbuffer. The actual crash occurs when XPutImage or XShmPutImage is called. At this point X11 will bail out when given an XImage with a depth of 32bit.

The fix? It's as simple as prepending this code to the above loop:

XWindowAttributes attribs;
if(XGetWindowAttributes(display,window,&attribs)) {
  if(XMatchVisualInfo(display, 
    DefaultScreen(display), 
    attribs.depth, 
    TrueColor, 
    &vi) {
      depth = attribs.depth;
      mode = TrueColor;
  }
} 
if(!depth){
  ...
}

Pretty much every distribution has complained about this issue over the past few years, so I am happy that we finally resolved this for good. Is this fix in the current beta of the Flash Player (9.0.21.55)? No, I just changed this Friday, so you'll have to wait for a newer build. It also needs to go through QE before this change gets the final approval.