html5

Queue & Cyclic Queue visualization with ProcessingJS

Previous posts:

Updates

I finished refactoring the code and loading all the scripts using requireJS.
RequireJS not only helped modularize the code but it also helped me find/fix a few bugs.

I knew about modular script loaders by working with nodeJS and commonJS
However, I had never used a script loader on the client side, and after playing with requireJS I can’t see myself not using anymore.
One of my last projects we didn’t anticipate the amount of client side code we would need to write, thus not planning and design a proper architecture. The code was mostly composed by event handlers and some logic needed to determine the flow of the app. It tried to mimic the functionality provided by BackboneJS and SpineJS
With that being said, once the app grew the event handling code started to get bloated and hard to maintain and some logic ended up being repeated across files.
RequireJS more than anything forces you to think in a modular way when creating web apps, and that helps maintain and also scale the app while it grows.

Here is an overview of the structure so far

The Element object is the tile that is used by the processing canvas to represent an element of a data structure
Since the Element can be reused between different data structures, it doesn’t contains any logic regarding how it should behave. It just knows where to position itself and to animate its entry and exit from the canvas.
That worked so far since all three data structures are some what similar. However, the idea is to have an abstract Element and derive separate elements for each data structure/algorithm. This would give more flexibility to each element once more examples are added.

Entry Point

RequireJS allows you to create an entry point for the application, where you can initialize any objects you need.

main.js

  
 require(  
 [  
 "./stack/stack",  
 "./queue/queue",  
 "./cyclic-queue/cyclic-queue",  
 ],  
 function () {

 // Init Stack  
 var Stack = require("./stack/stack");  
 var stack = new Stack();  
 stack.init();

 // Init Queue  
 var Queue = require("./queue/queue");  
 var queue = new Queue();  
 queue.init();

 // Init CyclicQueue  
 var CyclicQueue = require("./cyclic-queue/cyclic-queue");  
 var cyclicQueue = new CyclicQueue();  
 cyclicQueue.init();  
 });  

The first argument passed to require is a list of scripts to load before running the code.
The second argument is the code that will run once all the scripts specified in the first argument are loaded.

In the example above, the stack, queue and cyclic-queue scripts are loaded, then after their instantiation they are initialized.
All the logic of how they need to be initialized and how they will interact with the browser events are define within themselves, so to add a new data structure/algorithm the only thing that needs to be added to the main.js is the instantiation and initialization of the new object.
**An option for a future iteration is to create a factory to instantiate different objects, moving the responsibility of creating objects away from the main.js

Processing canvas

Since each example has its own canvas, a different instance of processing needs to be created.
However, not only the specific code for each example needs a reference to the processing canvas, but also the Element object, and right now all examples use the same Element.
Problem The Element object wasn’t referencing the right instance of processingJS

Solution:
Use requireJS to modularize the instantiation of the processingJS objects

So all the Elements will always reference the right canvas, and its code can be reused between the examples.
*A factory would probably be useful here to centralize the instantiation of the processing objects.

Some screen-shots:

You can find the code here and a live examples here

What’s Next?

There still a bunch of stuff to be done and improved.
The next iteration I plan to create a visualization for Hash Tables
I also want to refactor the code a little bit more and use factories to create processingJS canvas and to initialize the examples

References:

Bug 713608 Update

One of the bugs I’m currently working on is: Bug 713608 – HTML5 Video controls are missing in Fullscreen
A quick overview of the bug:
This bug involves working with html5 video controls.
The goal of the bug is to enable the stock controls when the video enters in fullscreen via the context menu. For example: the controls are hidden when the video isn’t in fullscreen, then the user right clicks on the video and selects enter fullscreen. When the video switches to fullscreen mode, the controls will be displayed, even though they weren’t before toggling fullscreen, and after exiting fullscreen mode, if the controls were disabled in the first place, disable them again.

After working on Bug 714071 – The Show Statistics setting is not preserved when toggling the full screen that also involved working on the videocontrols.xml file, I thought would be good to keep digging how XBL bindings work on firefox.
So far Bug 713608 has proven to be more complicated than I thought it would be.
At the beginning, I followed the same logic used on Bug 714071 and tried to find the method that shows/hides the controls. However, different from the Statistics menu option, it seems that hiding the controls uses a different approach.
This is the code that displays the statistics of a video

   
 showStatistics : function(shouldShow) {  
 if (this.statsInterval) {  
 clearInterval(this.statsInterval);  
 this.statsInterval = null;  
 }

 if (shouldShow) {  
 this.video.mozMediaStatisticsShowing = true;

 this.statsOverlay.hidden = false;  
 this.statsInterval = setInterval(this.updateStats.bind(this), this.STATS_INTERVAL_MS);  
 this.updateStats();  
 } else {  
 delete this.video.mozMediaStatisticsShowing;  
 this.statsOverlay.hidden = true;  
 }  
 },  

But I couldn’t find the equivalent method for the video controls.

Browsing through the code, I was able to find the methods triggered by the event listeners for mousemove/over/out used by the video element:

MouseMove

   
 onMouseMove : function (event) {  
 // If the controls are static, don’t change anything.  
 if (!this.dynamicControls)  
 return;

 clearTimeout(this._hideControlsTimeout);

 // Suppress fading out the controls until the video has rendered  
 // its first frame. But since autoplay videos start off with no  
 // controls, let them fade-out so the controls don’t get stuck on.  
 if (!this.firstFrameShown &&  
 !(this.video.autoplay && this.video.mozAutoplayEnabled))  
 return;

 this.startFade(this.controlBar, true);  
 // Hide the controls if the mouse cursor is left on top of the video  
 // but above the control bar and if the click-to-play overlay is hidden.  
 if (event.clientY < this.controlBar.getBoundingClientRect().top && this.clickToPlay.hidden) {  
 this._hideControlsTimeout = setTimeout(this._hideControlsFn, this.HIDE_CONTROLS_TIMEOUT_MS);  
 }  
 },  

MouseInOut

   
 onMouseInOut : function (event) {  
 // If the controls are static, don’t change anything.  
 if (!this.dynamicControls)  
 return;

 clearTimeout(this._hideControlsTimeout);

 // Ignore events caused by transitions between child nodes.  
 // Note that the videocontrols element is the same  
 // size as the *content area* of the video element,  
 // but this is not the same as the video element’s  
 // border area if the video has border or padding.  
 if (this.isEventWithin(event, this.videocontrols))  
 return;

 var isMouseOver = (event.type == "mouseover");

 // Suppress fading out the controls until the video has rendered  
 // its first frame. But since autoplay videos start off with no  
 // controls, let them fade-out so the controls don’t get stuck on.  
 if (!this.firstFrameShown && !isMouseOver &&  
 !(this.video.autoplay && this.video.mozAutoplayEnabled))  
 return;

 if (!isMouseOver) {  
 this.adjustControlSize();

 // Keep the controls visible if the click-to-play is visible.  
 if (!this.clickToPlay.hidden)  
 return;

 // Setting a timer here to handle the case where the mouse leaves  
 // the video from hovering over the controls.  
 this._hideControlsTimeout = setTimeout(this._hideControlsFn, this.HIDE_CONTROLS_TIMEOUT_MS);  
 }  
 },  

My plan was to find the event listeners that trigger the display of the video controls, then find how those events are added/removed to element.

I found where the listeners were defined

   
   
   
 if (!this.isTouchControl)  
 this.Utils.onMouseInOut(event);  
   
   
 if (!this.isTouchControl)  
 this.Utils.onMouseInOut(event);  
   
   
 if (!this.isTouchControl)  
 this.Utils.onMouseMove(event);  
   
   

However, I couldn’t find where they are added/removed from the video element.

I feel that I’m getting close to finding a solution.

Big thanks to Jared Wein (jaws) for helping me with this and other bugs :)

Firefox Bug 714071

Working on Bug 714071 introduced me to another layer of Firefox.
So far all I’ve been doing was working with c++ code, specifically related to the MouseLock API.
Bug 714071 on the other hand was focused on the js layer.

A brief summary of the bug:

Fix a problem with the Statistics video control


When the showing statistics option of a video was on, and the fullscreen was toggled the video would stop displaying the statistics but the menu would not be updated.
The goal for the bug was to keep showing the statistics when toggling between fullscreen.

The code that displays the statistics on a video is the following:

   
 showStatistics : function(shouldShow) {  
 if (this.statsInterval) {  
 clearInterval(this.statsInterval);  
 this.statsInterval = null;  
 }

 if (shouldShow) {  
 this.video.mozMediaStatisticsShowing = true;

 this.statsOverlay.hidden = false;  
 this.statsInterval = setInterval(this.updateStats.bind(this), this.STATS_INTERVAL_MS);  
 this.updateStats();  
 } else {  
 delete this.video.mozMediaStatisticsShowing;  
 this.statsOverlay.hidden = true;  
 }  
 },

Everytime the video was loaded on the page, or toggled in fullscreen, the init method would be called and an event listener would be attached to listen for the “Show Statistics” option click. However, the init method would initialize the video with a fresh config, so if the statistics were being displayed it would be hidden after switching between fullscreen mode.

Solution

The solution was to add a check on the setUpInitialState method to activate the statistics on the video if they were being displayed before toggling the fullscreen.

   
 if (this.video.mozMediaStatisticsShowing) {  
 this.showStatistics(true);  
 }  

That apparently solved the problem and the statistics are preserved even when toggling fullscreen.

Working on this bug made me very curious. How could some javascript code interact with c++ at run time.
I knew about the XPCOM object model used in Firefox and lately I’ve started to read more about XUL and the Gecko engine . This was the perfect time to start digging more deep and learn more about the Firefox foundations.
Using the videocontrols as a starting point I went to mxr and started searching some code.
From my initial search I think I found where the controls were loaded in c++. Now I just need to figure it out how all that happens :)

videoscontrols.xml – define the video controls

nsCOMPtr mVideoControls;
nsVideoFrame::CreateAnonymousContent – where the menu gets attached to the video element

nsNodeInfoManager – apparently used to load the video controls as well the poster image
nsNodeInfoMangager::GetNodeInfo – looks like the loading happens here

Manage XPCOM objects?
PLHasTableLookup
PLHasEntry
PL
HasTableAdd

NSTrustedNewXULElement – where the XUL element gets created and casted to an nsIContent
NS
TrustedNewXULElement declaration
NS_TrustedNewXULElement definition

TODO:
Keep digging and find how the pieces fit together!

Managing Game Assets - XHR2 and URLObjects

I ran into a problem while working on a ticket in the gladius project.
The ticket that I’m working on is about loading resources into the browser’s memory.
The part that I’m struggling with is how to remove the resources from the memory once they’ve been used.
For example:
Lets say a game has 10 levels, each level has different resources. It make sense to load the resources for level 1, level 2, etc, separately. So when the player finishes level 1 and goes to level 2, the resources from level 1 will be unloaded from memory.

I spent a few hours on google trying to find a way to manipulate the browser’s cache with javascript, but I hit a dead end. Not only I didn’t find any information, but all the information I found was saying that it was impossible to do such a thing.

if (!google) goto IRC :)

So there I was. Asking questions on different channels hoping to find a solution. The first answer I got wasn’t very encouraging. I was told that it was impossible to do what I wanted. But I knew it had to be possible. The ticket specifications were very clear, the module needed to unload cached content. So I kept asking.

Luckily, the second answer was a little bit more optimistic. It pointed me in the direction of XHR2 requests and URLObjects.

XHR2(XMLHttpRequest Level 2) has a lot of new features compared to the old XMLHttpRequest,some of the features are:
Cross-origin request
Upload progress events
Upload/Download of binary data
Plus, it can do a lot of other cools things, such as working with the new HTML5 APIs, like the File System, WebAudio and WebGL

But why XHR2 is handy when it comes to loading an image into the browser’s memory?
Because it provides a way to make a request to the server with a responseType set as blob. So the server sends the image back in binary format.
With the image in binary format it’s possible to create a temporary URLObject.
By having an image loaded as a URLObject, it gives the option to dynamically unload the image from the browser memory whenever needed. Here is an example:

 
window.URL = window.URL || window.webkitURL;`

function getBlob(url, cb) {
var r = new XMLHttpRequest();
r.open("GET", url);
r.responseType = "blob" // XHR2
r.send(null);
r.onload = function() { // XHR2
objectURL = window.URL.createObjectURL(r.response); // Creates a temporary URL```

console.log('objectURL: ' + objectURL); // moz-filedata:b42305a0-29a2-4b1e-ae02-6ef78b4cfe4e`

img = new Image();
img.src = objectURL;
img.onerror = function() {
console.log('onerror');
}
img.onload = function() {
console.log('onload');
}
}
}
.....
.....
getBlob('img1.jpg', null);
.....
.....
window.URL.revokeObjectURL(objectURL); // Releases the memory

Even though the solution worked, I’m not sure if it’s a feasible one.
First, it’s not supported by all major browsers. Only Chrome 8+ and Firefox 4+ provide support.
Second, I don’t know what are the performance issues compared to simple http requests.
However, using this technique it should be possible to implement webworkers to make the requests and then post back a message to the page containing the binary for the resource.(*it’s not)!
*
At the moment XHR2 requests are not supported in a worker thread
The bug is already filled in bugzilla
https://bugzilla.mozilla.org/showbug.cgi?id=675504
https://bugzilla.mozilla.org/show
bug.cgi?id=658178

If anybody is interested on the topic, these are some good references:

http://www.html5rocks.com/en/tutorials/file/xhr2/
http://caniuse.com/xhr2
http://davidflanagan.com/Talks/jsconf11/BytesAndBlobs.html
https://developer.mozilla.org/en/DOM/window.URL.createObjectURL
https://developer.mozilla.org/en/DOM/window.URL.revokeObjectURL#Browsercompatibility
https://developer.mozilla.org/en/Using
filesfromwebapplications#Example.3aUsingobjectURLstodisplay_images
http://www.htmlfivewow.com/slide45
http://dev.w3.org/2006/webapi/FileAPI/
http://www.opera.com/docs/specs/presto28/file/