More than GMapbook: 2006

I've decided to implement a weather map in blogs that supports javascript. Actually, I have already make one, but it works only in IE, never in Firfox, and I got an error message like this "Error: uncaught exception: Permission denied to call method XMLHttpRequest.open".

Firefox diables XMLHttpRequest to files on a different domain from the original one for security reason. See also: http://www.w3.org/TR/XMLHttpRequest/#security
Sigh, I don't want to ask firefox users to change their security settings so as to visit my site, but I find an interesting article on fettig.net. Abe Fettig's way to solve this problem might be useful to be. See below(How to make XmlHttpRequest calls to another server in your domain):

This post is more technical and detailed than what I usually write on fettig.net. I’m going to talk about a technique I’ve been working on to work around a limitation in the XmlHttpRequest object used in Ajax applications. Web geeks, please read on.

Updates to this post

See this post for an updated version of this techinique that works (hack-free) in Firefox 1.5.

The problem

XmlHttpRequest objects are bound by the same origin security policy of browsers, which prevents a page from accessing data from another server. This has put a serious limitation on Ajax developers: you can use XmlHttpRequests to make background calls to a server, but it has to be the same server that served up the current page. Known workarounds for this limitation involve either server-side reverse proxying or bypassing XmlHttpRequest entirely.

In my case neither of these approaches was going to work. I wanted to use LivePage, the live-update framework developed by Donovan Preston and other brilliant hackers at Divmod. LivePage works because it uses Twisted, which is good at handling lots of long-lasting network connections at the same time. Since Apache isn’t good at handling lots of long-lasting connections, putting an Apache reverse proxy in front of the Twisted server would put a major hurt on performance and scalability. And since LivePage is bound to XmlHttpRequest, I couldn’t use a non-XmlHttpRequest alternative.

Our approach with JotSpot Live has been to let a Twisted server handle all page requests and XmlHttpRequest calls. That’s fine as long as JotSpot Live is a standalone, dedicated web site. But what our customers are asking for (and what we want) is the ability to have Live-style realtime updates in ordinary xxx.jot.com sites. And we don’t want to put a Twisted server in front of every site in our domain. So I’ve been trying to find a way to let any page in our domain communicate with live.jot.com through XmlHttpRequests. As it turns out, it is possible, but you have to jump through some hoops.

A couple of notes on the examples here:

I’m not really using two different servers. In my examples, a page on http://fettig.net/ is attempting to make an XmlHttpRequest call to http://www.fettig.net/. The fact that my Apache setup maps these to the same virtual host makes no difference to a browser, though: they’re not the same hostname, and they’re treated no differently than if they were two different sites with different content. The results would be the same if I was trying to make XmlHttpRequest calls to an ajax.fettig.net server with its own IP address.

I’m using a simple XmlHttpRequest wrapper library that provides a getUrl function. getUrl takes a URL and a callback function, opens an XmlHttpRequest connection to that URL, and calls the function with the results. If you’re interested you can view the full code here.

The service to which I’m making XmlHttpRequest calls is a simple PHP page, ajaxdata.php, that prints out the current UNIX time on the server: Not particularly useful in real life, but good enough for testing purposes.

First attempt: the naive approach

Here’s the first thing I tried: a standard XmlHttpRequest with the full URL of a page in a different subdomain. I was pretty sure this wouldn’t work, but I wanted to verify it for myself:
<html>
 <head>
 <script type=”text/javascript” src=”xmlhttp.js”></script>
 <script type=”text/javascript”>
  var AJAX_URL=”http://www.fettig.net/playground/ajax-subdomain/ajaxdata.php”;
  function getTime(){
     getUrl(AJAX_URL, gotTime);
   }
   function gotTime(status, headers, result) {
     document.getElementById(’time’).innerHTML = result;
     setTimeout(getTime, 1000)
   }
   window.onload = getTime;
 </script>
 </head>
 <body>
   <div id=”time”></div>
 </body>
</html>
This page is served from fettig.net, and tries to make an XmlHttpRequest call to www.fettig.net. In real life, these are the same server, but the browser doesn’t know that. Unsurprisingly, this doesn’t work in any browser. You get a security error such as this one (from Firefox):
Error: uncaught exception: Permission denied to call method XMLHttpRequest.open
You can try this for yourself here.

Second attempt: using an iframe and document.domain

On my last trip to the JotSpot office Alex pointed me toward iframes and the document.domain property. iframes are similar to XmlHttpRequests in that they load data into the current page from another page. But iframes aren’t limited to pulling pages from same web server as their containing page - they can load any URL. To prevent cross-site security problems, browsers enforce the same origin policy in the javascript object model: scripts running in one frame can’t access any objects inside another iframe, unless both pages came from the same server.

There’s an exception to this rule, however. If both pages come from the same parent domain, and both of them set the property document.domain to the same parent domain, scripts running in either frame will be allowed to talk to each other. For example, say the page http://www.example.com/ loads the page http://ajax.example.com/ in an iframe. Since both pages are in the domain example.com, if both set document.domain to “example.com” they will be be given the ability to programatically access each other’s data. While there are a few differences

So, can you use an iframe with document.domain to make XmlHttpRequest connections? Yes, with two restrictions:

The iframe must be served from the server to which you’ll be making XmlHttpRequest calls.

You have to open the XmlHttpRequest connection before you set document.domain.

Here’s the code I used. First, the page test2.html:
<html>
 <head>
 <script type=”text/javascript”>
  document.domain=”fettig.net”;
  function gotTime(result) {
    document.getElementById(’time’).innerHTML = “Server timestamp: ” + result;
  }
 </script>
 </head>
 <body>
   Single XmlHttpRequest. Works in all modern browsers.
   <div id=”time”></div>
   <iframe src=”http://www.fettig.net/playground/ajax-subdomain/test2-iframe.html”>
   </iframe>
 </body>
</html>
This script in this page sets document.domain, and defines the function gotTime to handle the results of the XmlHttpRequest call. All the XmlHttpRequest stuff happens in an iframe, though. Here’s the code in test2-iframe.html:
<html>
 <head>
 <script type=”text/javascript” src=”xmlhttp.js”></script>
 <script type=”text/javascript”>
   var AJAX_URL=”http://www.fettig.net/playground/ajax-subdomain/ajaxdata.php”;
   function gotResult(status, headers, result) {
     document.domain = “fettig.net”; // set d.d before talking to parent frame
     window.parent.gotTime(result);
   }
   getUrl(AJAX_URL, gotResult);
 </script>
 </head>
</html>
test2-iframe.html makes the XmlHttpRequest call to www.fettig.net. It’s allowed to do this because it was served from www.fettig.net, so it’s just talking to its originating server. Once it gets the response back, it sets document.domain to fettig.net, to match the document.domain value of the parent frame. Now it’s allowed to talk to the parent frame using javascript, so it’s able to call window.parent.gotTime.

The key is to do this in the right order. Once you set document.domain (to anything other then the actual host the page was served from) you lose the ability to make XmlHttpRequest calls. So you’ve got to take care of your XmlHttpRequest business before you set document.domain and to enable communicating with the parent frame.

Follow those rules, though, and you’re good to go. This technique seems to work in all modern browsers (I tested in IE 6, Firefox 1.0.7, Safari 1.3, Opera 8.5, and Konqueror 3.4), and it doesn’t contain anything I’d consider a hack. You can try it yourself here. So far so good!

Third attempt: repeated XmlHttpRequests

I was happy to have figured out a way to make a single XmlHttpRequest to another server in our domain. But as presented so far, this technique has a pretty severe limitation: You can only make XmlHttpRequest calls up to the point where you set document.domain. Once you do that, you gain the ability to communicate with the parent frame, but you lose the ability to make future XmlHttpRequest calls. This wasn’t good enough for LivePage, which needs the ability to continuously make XmlHttpRequest calls to the server and handle the results. I needed a way to let the iframe communicate with both the server (using XmlHttpRequests) and the parent frame (which it can only do by setting document.domain).

This raised the question: is setting the document.domain property one-time-only, or is it possible to switch it on the fly? If it can be changed back and forth, it might be possible to change it back and forth as needed. Here’s test3-iframe.html, a modified version of the iframe page:
<html>
 <head>
 <script type=”text/javascript” src=”xmlhttp.js”></script>
 <script type=”text/javascript”>
   var AJAX_URL=”http://www.fettig.net/playground/ajax-subdomain/ajaxdata.php”;
   function gotResult(status, headers, result) {
     var oldDomain = document.domain;
     document.domain = “fettig.net”;
     window.parent.gotTime(result);
     document.domain = oldDomain;
     setTimeout(getTime, 1000);
   }
   function getTime(){
     getUrl(AJAX_URL, gotResult);
   }
   getTime();
 </script>
 </head>
</html>
Like in test2-iframe.html, the gotResult function in test3-iframe.html sets document.domain to fettig.net (to enable access to the parent frame) and then calls window.parent.gotTime. But first it stores the current value of document.domain, which defaults to the host the page was served from. After it’s finished working with the parent frame, it sets document.domain back to the original value, which restores the ability to make XmlHttpRequests. It then uses setTimeout to set up another XmlHttpRequest call. The result should be a continually updated timestamp on the main page.

You can see this in action at http://fettig.net/playground/ajax-subdomain/test3.html . This page makes repeated XmlHttpRequests, with the iframe calling a function in the parent page each time it gets a result from the server. Unfortunately, while this technique works great in IE, Safari, and Konqueror, it doesn’t work in Mozilla and Opera. The first XmlHttpRequest will work, but when you try to change document.domain back to the original value you get an error. From Firefox:
Error: [Exception... "Illegal document.domain value"
code: “1009″
nsresult: “0×805303f1 (NS_ERROR_DOM_BAD_DOCUMENT_DOMAIN)”
location: “http://www.fettig.net/playground/ajax-subdomain/test3-iframe.html
Line: 13″]
Mozilla and Opera are more strict about the value of document.domain - it can only be set to the current value or a higher-level domain. For example, if the host is aaa.bbb.example.com, I could set document.domain to bbb.example.com. At that point I could change it again, to example.com, but I couldn’t change it back to aaa.bbb.example.com. Once you move to a higher-level domain, you’re stuck there.

Fourth attempt: Making it work in Mozilla

I could live without Opera, but not Mozilla. So I set out to find a way - any way - to work around this problem. After thinking about it for a while I started wondering: just how strict are security limitations on communications between frames? We know that browsers don’t let frames access each other’s objects unless they have the same document.domain. So setting a child frame’s document.domain to something different from the parent’s and then trying to look at window.parent.foo isn’t going to work. But what if you could set an attribute in the child frame that pointed to a function in the parent frame, and then changed the document domain? Would the child frame still be able to call that function?

The answer is yes, and that was the break I needed to get this thing working in Mozilla. The trick is to use two frames:
parent window
 bridge iframe
   child iframe
The bridge iframe and child iframe are both served from the server to which you’ll be making XmlHttpRequest calls. The parent page code is the same as in the previous examples: it defines a function to handle the XmlHttpRequest result, and loads an iframe, this time containing the bridge code, which looks like this:
<html>
 <head>
 <script type=”text/javascript” src=”xmlhttp.js”></script>
 <script type=”text/javascript”>
   function gotTime(result) {
     window.parent.gotTime(result); // pass result up to the parent
   }
   window.onload = function(){
     var subframe = document.createElement(’iframe’);
     document.body.appendChild(subframe);
     subframe.src = “test4-iframe.html”;
     subframe.contentWindow.bridgeGotTime = gotTime;
     document.domain = “fettig.net”;
   }
 </script>
 </head>
 <body></body>
</html>
When this frame loads, it loads another frame inside itself. Since the child frame is loading from the same server as the bridge frame, the bridge frame can access the child frame’s objects through script. The bridge frame takes advantage of this to set subframe.contentWindow.bridgeGotTime to its own gotTime function. Next, the bridge frame changes its document.domain to match that of the parent window. At this point the bridge frame loses the ability to talk directly to the child frame, since their document.domain properties no longer match. But the child frame retains the ability to communicate with the bridge frame through the bridgeGotTime function, which was set up first. And since the bridge frame’s document.domain now matches the parent frame’s document.domain, the bridge and parent can communicate freely.

In my limited testing, this techinique only seems to work on Mozilla-based browsers (I haven’t found a solution for Opera yet, but I haven’t spent much time on it). My final example uses a hybrid approach that tries to use document.domain switching, and then falls back on the bridge-iframe approach if the browser raises an error when you try to reset document.domain. Here’s the main page code:
<html>
 <head>
 <script type=”text/javascript”>
   document.domain=”fettig.net”;
   function gotTime(result) {
     document.getElementById(’time’).innerHTML = “Server timestamp: ” + result;    }
 </script>
 </head>
 <body>
   <div id=”time”></div>
   <iframe src=”http://www.fettig.net/playground/ajax-subdomain/test4-iframe.html”></iframe>
 </body>
</html>
Same as everything else so far except that the iframe is loaded from test4-iframe.html. Here’s the code in that page:
<html>
 <head>
 <script type=”text/javascript” src=”xmlhttp.js”></script>
 <script type=”text/javascript”>
   var AJAX_URL=”http://www.fettig.net/playground/ajax-subdomain/ajaxdata.php”;    function getTime(){
     getUrl(AJAX_URL, gotTime);
   }
   function gotTime(status, headers, result) {
     var oldDomain = document.domain;
     if (window.bridgeGotTime) {
       window.bridgeGotTime(result);
     } else {
       document.domain = “fettig.net”;
       window.parent.gotTime(result);
     }
     try {
       document.domain = oldDomain;
       setTimeout(getTime, 1000);
     } catch(e) {
       // denied access to switching the domain, use bridge instead
       document.location.replace(”test4-bridge.html”);
     }
   }
   getTime();
 </script>
 </head>
</html>
This code tries to switch document.domain back and forth as needed to communicate with both the XmlHttpRequest server and parent frame. If this fails, however, it loads the bridge iframe in its place. To keep from having two different pages doing the same thing, this page also acts as the child of the bridge frame. If it sees a window.bridgeGotTime attribute it knows it’s running under the bridge iframe, so it calls window.bridgeGotTime instead of trying to talk directly to window.parent.

You can see this in action in test4.html. It works in all the latest browsers except Opera. Update:use test5.html instead. See this post for details.

End result: it’s possible to make XmlHttpRequest calls to another server in your domain. You can make repeated calls too, if you’re willing to deal with a little setup overhead. This has a number of possible benefits:

You could have a dedicated Twisted server handling LivePage requests, while the main content of pages if served by Apache or some other web server.

You could have a dedicated server for handling web services, including XmlHttpRequests

You can use DNS to free XmlHttpRequest from the two-max-connections limit of browsers. Browsers don’t let you have more then two open connections to a server at a time - this gums things up if you’re using an app like JotSpot Live and try to have more then one tab going at a time. But if the XmlHttpRequests don’t have to be going to the same server, you can use wildcard DNS and set each page to make XmlHttpRequest calls something like randomhash.livepage.yourdomain.com.

If you’re read this far, thanks for hanging in there! I hope this is a useful tool in your bag of Ajax tricks. I’m sure there’s more to be explored here, but having gotten this far I wanted to write this post and put it up for comments. If you think I’ve overlooked something, that there’s a better way, or that all of this is an ugly hack, please tell me about it in the comments.

More than GMapbook

Thursday 14 December 2006

GMapbook doesn't support IE7 well

Monday 4 December 2006

About "Error: uncaught exception: Permission denied to call method XMLHttpRequest.open"

Updates to this post

The problem

First attempt: the naive approach

Second attempt: using an iframe and document.domain

Third attempt: repeated XmlHttpRequests

Fourth attempt: Making it work in Mozilla

Saturday 2 December 2006

What to do next

Friday 1 December 2006

GMapbook v1.0 released!