Getting the mouse to unlock whenever the window loses fullscreen mode wasn’t an easy task.
Mostly because I’m not very familiar with the firefox code, and I’m still trying to understand how all the different components are able to communicate between them.
A few times I thought about giving up on the task, but I’m glad I didn’t. Looking back, the amount of new things I learnt was worth every minute spent.

1. Finding where to write the code

This is the call call stack for a fullscreen request:

I just went as low as the nsWindow::MakeFullScreen, since there is where the call to native libraries happens.
Because the nsWindow is platform specific, I decided to go one level of above. To the nsGlobalWindow::SetFullScreen

The SetFullScreen method of the nsGlobalWindow is called whenever the status of the screen needs to be changed(to enter or exit fullscreen).
I thought that placing the code to unlock the mouse on the SetFullScreen would be a good option, since it’s the last method call before going into platform specific files.

2. The problem

At first I thought that if the screen exited fullscreen mode the nsGlobalWindow::SetFullScreen would be called, then I would get a reference to the navigator, then to the pointer, and then unlock the mouse.
Here is the stack for canceling the fullscreen
Cancel fullscreen stack

On the stack, I saw that the nsDocument::MozCancelFullScreen was called, then the nsDocument::CancelFullScreen, and then kept going until the nsGlobalWindow::SetFullScreen

The only problem was that when the window exited the fullscreen mode, the nsGlobalWindow:SetFullScreen would only set the fullscreen status of the root window. However, the navigator was attached to the outer window, so it wasn’t possible to access the navigator when the fullscreen mode was canceled.

Different memory address.

3. Inspecting the problem

I saved some printf’s

First I opened the js console an accessed the navigator by typing window.navigator
Accessing window.navigator from js console

Line 7 gets the navigator for window7
Line 17 creates the navigator, since its the first time its being referred to

Then on the same page I requested an element to go fullscreen, and later canceled the fullscreen
Requesting and Canceling fullscreen

Line 7 is setting the window7 to fullscreen (the outer window, same as the one that created the navigator
Line 11 checks to see if window7 is the root window, because isn’t, it calls SetFullScreen on the root window, that has an id of 1

Line 20 the document is canceling the fullscreen
Line 21 it’s the id of the window of the document that is being canceled.
Line 29 is existing fullscreen mode of window1
Line 37 gets the navigator for the window1
Line 57 unlocks the mouse for window1

When the window exited the fullscreen mode, it was window1(root) that was set to false, not the window7(outer), and because each window has a different navigator, the navigator for window1 wasn’t the same as the one for window7

4. The solution

My solution, was to write the code to unlock the mouse on nsDocument::CancelFullScreen. On the CancelFullScreen method, there is a loop that iterates through the documents, and sets their fullscreen status to false. Inside the loop, one of the iterations gives the document which has the locked element. So with a reference to the doc, it’s possible to get the window for that document, the navigator, the pointer, and finally, unlock it.

 while (doc != nsnull) {  
 if (::SetFullScreenState(doc, nsnull, false)) {

 nsCOMPtr window = doc->GetWindow();  
 nsCOMPtr navigator;  

 if (navigator) {  
 nsCOMPtr pointer;  
 if (pointer) {  

 doc = doc->GetParentDocument();  

Now whenever the element loses fullscreen mode, the mouse is automatically unlocked.

*The only reason I got this working, was because of the great help of the mozilla community.
Big thanks to cpearce!