After reading the MouseLock wiki page I thought that the option 8 “Mouse lock should only work when in Full Screen Mode” would be a good place to start hacking.

My first searches led me to the GetFullScreen method of nsGlobalWindow http://mxr.mozilla.org/mozilla-central/source/dom/base/nsGlobalWindow.cpp#4453

Knowing that to get the fullscreen status of a browser window I would need to use the nsGlobalWindow::GetFullScreen the next step was to find how to call that method from the MouseLock class. I struggled a bit with that, since to get the FullScreen status of the window I would first need to have access to the window itself. After hitting a dead wall I went to irc looking for some help. Luckly, David Humphrey was there and was able to provide some good explanations for my problem. He suggested to create an Init method on MouseLock, similar to what the Geolocation class is doing.

After a good amount of hours and a good amount of errors….it worked! :)

So what happens is that on the Navigator::GetPointer, a MouseLock object is created and the Init method is called passing a reference to the current window as an argument.
The MouseLock::Init stores the window reference as a member variable, so then all the other methods can access the current window and check for its fullscreen status.

   
 nsresult  
 nsDOMMouseLockable::Init(nsIDOMWindow* aContentWindow)  
 {  
 NS_ENSURE_ARG_POINTER(aContentWindow);  
 mWindow = aContentWindow;  
 return NS_OK;  
 }

NS_IMETHODIMP nsDOMMouseLockable::Islocked(bool *_retval NS_OUTPARAM)  
 {  
 // Check the status of the window  
 bool* isFullScreen = new bool;  
 mWindow->GetFullScreen(isFullScreen);  
 printf("nisFullScreen? %sn", *isFullScreen ? "true" : "false");  
 *_retval = mIsLocked;  
 return NS_OK;  
 }  
   
 NS_IMETHODIMP  
 Navigator::GetPointer(nsIDOMMouseLockable** aPointer)  
 {  
 NS_ENSURE_ARG_POINTER(aPointer);

 if (!mPointer) {  
 mPointer = new nsDOMMouseLockable();  
 }

 if (!mDocShell) {  
 return NS_ERROR_FAILURE;  
 }

 nsCOMPtr contentDOMWindow = do_GetInterface(mDocShell);  
 if (!contentDOMWindow) {  
 return NS_ERROR_FAILURE;  
 }

 if (NS_FAILED(mPointer->Init(contentDOMWindow))) {  
 mPointer = nsnull;  
 return NS_ERROR_FAILURE;  
 }

 NS_ADDREF(*aPointer = mPointer);  
 return NS_OK;  
 }  

I committed all the changes on github

Here are a couple of screenshots testing if the MouseLock was getting the right values of the fullscreen status: