XMLHttpRequest and readyState==3
We are in the process of trying to make SynchroEdit <www.synchroedit.com> work with Safari 3.0, with the eventual goal of seeing if it is possible to synchroedit documents using an iPhone. SynchroEdit is an open-source simultaneous web editor (in the style of SubEthaEdit) that currently only works in Mozilla/Firefox/Camino browsers. The technique it uses is to synchronize the DOM tree between multiple machines, so it potentially has some interesting uses beyond synchronous editing. For the most part, things seem to be working well, but one feature we make use of heavily is the readyState==3 effect which is supported by Firefox. We have read that this feature is unsupported by the WHATWG spec, and that it will most likely not be implemented into Safari. The reason why we need this functionality is that we are building highly-interactive web applications, and we have found that we need to cut down on the number of connections to and from the HTTP server, in order to gain performance. Opening a new connection for every input/output request would slow the browser, and the server, down substantially. As the browser is the initiator of all requests, pending output (from server to client) requests can only be determined by repeatedly polling the server for updates, or by keeping some form of streaming connection using a technique like the readystate technique as described below. Basically, the Firefox technique involves reading the <XMLHttpRequest>.responseText value while the request itself stays in readyState==3 (loading). The responseText buffer updates as new input appears, and in this manner a "streaming data connection" is achieved. Whenever new data is sent to the browser from the server via the XMLHttpRequest, Firefox triggers another "readystatechange", even though the readyState changes "from 3 to 3". This lets coders put in code akin to "onRead" for a regular TCP connection interface, but in AJAX apps. This technique has been hugely effective not only for SynchroEdit, but for other applications that keep a continuous connection to the server such as interactive chat applications. As an alternative approach that might work with WebKit, which we've tested in r23558, is to instead make a setInterval() "check_data" style function, which simply checks the size of the responseText string for updates. Example code* would look like: var rt_index = 0; var req = null; function itv_check_data() { var curr_index = req.responseText.length; if (rt_index == curr_index) return; // no new data var s = req.responseText.substring(rt_index); rt_index = curr_index; alert("new input: " + s); } function start_reading() { req = new_XHR(); // new XMLHttpRequest() in FF req.open("POST", "foo.cgi", true); req.send("test"); setInterval("itv_check_data()", 333); // run itv_check_data thrice/sec } * The above code is untested and may be bugged. This alternative technique worked fine in Firefox (but had a higher latency then the readystate==3 technique), but it does not seem to work in WebKit r23558 either. It appears WebKit is not setting the responseText variable until the request has completed. Is there a way to access this text prior to state 4? Another possible approach which we are investigating is to close the connection server-side, whenever new data is available. The connection is immediately reopened and stays open until new data is available or until the connection times out. This would remove the need for readyState==3 but may have too large an impact on performance. Any suggestions or ideas for us? -- Christopher Allen P.S. We think SynchroEdit would be an interested test-bed for WebKit as it makes extensive use of designMode-enabled frames and our the DOM synchronization tends to expose bugs in DOM. Let us know if you are interested in helping us port it to support WebKit/Safari.
For the most part, things seem to be working well, but one feature we make use of heavily is the readyState==3 effect which is supported by Firefox. We have read that this feature is unsupported by the WHATWG spec, and that it will most likely not be implemented into Safari.
Where did you hear that? From code inspection, I see that XMLHttpRequest issues a "ready state change" of 3 every time it receives data. Can you point me to the relevant part of the WHATWG spec?
It appears WebKit is not setting the responseText variable until the request has completed. Is there a way to access this text prior to state 4?
From code inspection, I see that XMLHttpRequest updates responseText every time it receives data. Perhaps you're seeing the results of slightly different networking implementations. For example, you might need to send data in larger chunks in order to convince the networking layer to flush its data up to the application level. Geoff
Geoffrey Garen wrote:
From code inspection, I see that XMLHttpRequest updates responseText every time it receives data.
Perhaps you're seeing the results of slightly different networking implementations. For example, you might need to send data in larger chunks in order to convince the networking layer to flush its data up to the application level.
We've written up a simple example using perl cgi and js/xhr to demonstrate this problem. In Firefox, the expected behavior appears. Every second, a new line appears counting up from 1 to 10. In Safari, there's a 10 second delay with nothing, and then 10 lines appear counting up from 1 to 10 all at once, just as the readyState goes from 3 to 4. We never see readyState going from 3 to 3 as we do with Firefox. URL demonstrating this behavior: http://www.synchroedit.com/pt/ URL with source code for this test: http://www.synchroedit.com/pt/perl-test.tar.gz The perl cgi script has autoflushing enabled, which means the buffer is flushed for every newline. In Firefox, the responseText is updated whenever the perl side sends and flushes data, while Safari's responseText stays empty until the perl cgi closes the connection and the request enters readyState 4. So to return back to our original question -- this capability is needed for synchronous applications that keep an ongoing connection to the server to avoid polling and other performance issues. Is this capability supposed to be to in Safari, thus our test above demonstrates a bug? Are we missing something? Is there an alternative approach that would make this behavior work right in WebKit/Safari 3 as well? Thanks for your help! -- Christopher Allen
On Jun 26, 2007, at 10:29 AM, Christopher Allen wrote:
Geoffrey Garen wrote:
From code inspection, I see that XMLHttpRequest updates responseText every time it receives data.
Perhaps you're seeing the results of slightly different networking implementations. For example, you might need to send data in larger chunks in order to convince the networking layer to flush its data up to the application level.
We've written up a simple example using perl cgi and js/xhr to demonstrate this problem. In Firefox, the expected behavior appears. Every second, a new line appears counting up from 1 to 10. In Safari, there's a 10 second delay with nothing, and then 10 lines appear counting up from 1 to 10 all at once, just as the readyState goes from 3 to 4. We never see readyState going from 3 to 3 as we do with Firefox.
URL demonstrating this behavior: http://www.synchroedit.com/pt/
URL with source code for this test: http://www.synchroedit.com/pt/perl-test.tar.gz
The perl cgi script has autoflushing enabled, which means the buffer is flushed for every newline. In Firefox, the responseText is updated whenever the perl side sends and flushes data, while Safari's responseText stays empty until the perl cgi closes the connection and the request enters readyState 4.
So to return back to our original question -- this capability is needed for synchronous applications that keep an ongoing connection to the server to avoid polling and other performance issues. Is this capability supposed to be to in Safari, thus our test above demonstrates a bug? Are we missing something? Is there an alternative approach that would make this behavior work right in WebKit/Safari 3 as well?
This is a networking layer issue - it buffers the data up to some limit depending on what MIME type you send it with. Two workarounds that I think will work: 1) "prime" the connection with 256 bytes sent before any of the real data. 2) Use a MIME type that won't be subject to sniffing (I think "application/xml" as opposed to "text/xml" may fit the bill). Regards, Maciej
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Maciej Stachowiak wrote:
This is a networking layer issue - it buffers the data up to some limit depending on what MIME type you send it with. Two workarounds that I think will work:
1) "prime" the connection with 256 bytes sent before any of the real data. 2) Use a MIME type that won't be subject to sniffing (I think "application/xml" as opposed to "text/xml" may fit the bill).
Thanks for the ideas! We tried both of these fixes (separately and in combination), but unless we're doing something wrong, neither is actually working 100%: - - In FF/linux, they all work. - - In Safari/windows, neither works. - - In Safari/Mac OS X, the 1+2 worked a little, but still missed some input. The URL for 1+2: http://www.synchroedit.com/pf1/ The URL for 2 only: http://www.synchroedit.com/pf2/ Sources 1+2: http://www.synchroedit.com/pf1/perl-fix-1.tar.gz Sources 2: http://www.synchroedit.com/pf2/perl-fix-2.tar.gz The first solution would most likely work if we increased the "garbage" from 256 chars to something like 512, but the overhead at 256 is already at something akin to 300% "more data" in transit, which sounds like a major performance loss. If at all possible, getting things to work without priming the connection would be ideal. Any ideas? - -Kalle. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGgnoGdNQyXs/kj34RAi8PAJ4rfal6Fh9M4sQa6KLa3E28GnDWCQCffcvJ BSZD/iRbZnqioB3wEDBjGDA= =yV9f -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Sorry for spamming the list, but thought I'd add that I've filed this as a bug here: http://bugs.webkit.org/show_bug.cgi?id=14431 - -Kalle. Kalle Alm wrote:
Maciej Stachowiak wrote:
This is a networking layer issue - it buffers the data up to some limit depending on what MIME type you send it with. Two workarounds that I think will work:
1) "prime" the connection with 256 bytes sent before any of the real data. 2) Use a MIME type that won't be subject to sniffing (I think "application/xml" as opposed to "text/xml" may fit the bill).
Thanks for the ideas! We tried both of these fixes (separately and in combination), but unless we're doing something wrong, neither is actually working 100%: - In FF/linux, they all work. - In Safari/windows, neither works. - In Safari/Mac OS X, the 1+2 worked a little, but still missed some input.
The URL for 1+2: http://www.synchroedit.com/pf1/ The URL for 2 only: http://www.synchroedit.com/pf2/ Sources 1+2: http://www.synchroedit.com/pf1/perl-fix-1.tar.gz Sources 2: http://www.synchroedit.com/pf2/perl-fix-2.tar.gz
The first solution would most likely work if we increased the "garbage" from 256 chars to something like 512, but the overhead at 256 is already at something akin to 300% "more data" in transit, which sounds like a major performance loss. If at all possible, getting things to work without priming the connection would be ideal.
Any ideas?
-Kalle.
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo/webkit-dev -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGgnwwdNQyXs/kj34RApajAJ46HjDExd9dkdU5IPYzIqiuebhlJACfcBe1 5FLFKAaFmqard8c2FngI96I= =hR1E -----END PGP SIGNATURE-----
Finally got a chance to test this for this Safari bug to see if it also exists on the iPhone, and to my surprise it is working perfectly there, almost exactly like Firefox: Firefox 2.0.0.4: run_test()... Opening request to poller.cgi (POST; asynchronous) Setting onreadystatechange handler Sending request State change: 0 => 1 Request was sent. State change: 1 => 2 State change: 2 => 3 Input: I am now 0 out of 10. State change: 3 => 3 Input: I am now 1 out of 10. State change: 3 => 3 Input: I am now 2 out of 10. State change: 3 => 3 Input: I am now 3 out of 10. State change: 3 => 3 Input: I am now 4 out of 10. State change: 3 => 3 Input: I am now 5 out of 10. State change: 3 => 3 Input: I am now 6 out of 10. State change: 3 => 3 Input: I am now 7 out of 10. State change: 3 => 3 Input: I am now 8 out of 10. State change: 3 => 3 Input: I am now 9 out of 10. State change: 3 => 3 Input: I am now 10 out of 10. State change: 3 => 4 iPhone 1.0 (: run_test()... Opening request to poller.cgi (POST; asynchronous) Setting onreadystatechange handler Sending request Request was sent State change: 0 => 2 State change: 2 => 3 Input: I am now 0 out of 10. State change: 3 => 3 Input: I am now 1 out of 10. State change: 3 => 3 Input: I am now 2 out of 10. State change: 3 => 3 Input: I am now 3 out of 10. State change: 3 => 3 Input: I am now 4 out of 10. State change: 3 => 3 Input: I am now 5 out of 10. State change: 3 => 3 Input: I am now 6 out of 10. State change: 3 => 3 Input: I am now 7 out of 10. State change: 3 => 3 Input: I am now 8 out of 10. State change: 3 => 3 Input: I am now 9 out of 10. State change: 3 => 3 Input: I am now 10 out of 10. State change: 3 => 4 I'm not sure why the trivial difference in the beginning, but we are quite pleased that even if this doesn't work on the desktop quite right, that it does work correctly on the iPhone. I'll have Kalle update the both bug reports: http://bugs.webkit.org/show_bug.cgi?id=14431 and Apple RADAR Problem ID is 5298296. Onward to some sophisticated iPhone AJAX webapps ;-) -- Christopher Allen For those of you interested, here is the history: On 6/25/07, Christopher Allen <ChristopherA@iphonewebdev.com> wrote:
We are in the process of trying to make SynchroEdit <www.synchroedit.com> work with Safari 3.0, with the eventual goal of seeing if it is possible to synchroedit documents using an iPhone.
SynchroEdit is an open-source simultaneous web editor (in the style of SubEthaEdit) that currently only works in Mozilla/Firefox/Camino browsers. The technique it uses is to synchronize the DOM tree between multiple machines, so it potentially has some interesting uses beyond synchronous editing.
For the most part, things seem to be working well, but one feature we make use of heavily is the readyState==3 effect which is supported by Firefox. We have read that this feature is unsupported by the WHATWG spec, and that it will most likely not be implemented into Safari.
The reason why we need this functionality is that we are building highly-interactive web applications, and we have found that we need to cut down on the number of connections to and from the HTTP server, in order to gain performance. Opening a new connection for every input/output request would slow the browser, and the server, down substantially. As the browser is the initiator of all requests, pending output (from server to client) requests can only be determined by repeatedly polling the server for updates, or by keeping some form of streaming connection using a technique like the readystate technique as described below.
Basically, the Firefox technique involves reading the <XMLHttpRequest>.responseText value while the request itself stays in readyState==3 (loading). The responseText buffer updates as new input appears, and in this manner a "streaming data connection" is achieved.
Whenever new data is sent to the browser from the server via the XMLHttpRequest, Firefox triggers another "readystatechange", even though the readyState changes "from 3 to 3". This lets coders put in code akin to "onRead" for a regular TCP connection interface, but in AJAX apps. This technique has been hugely effective not only for SynchroEdit, but for other applications that keep a continuous connection to the server such as interactive chat applications.
On 6/25/07, Geoffrey Garen <ggaren@apple.com> wrote:
For the most part, things seem to be working well, but one feature we make use of heavily is the readyState==3 effect which is supported by Firefox. We have read that this feature is unsupported by the WHATWG spec, and that it will most likely not be implemented into Safari.
Where did you hear that? From code inspection, I see that XMLHttpRequest issues a "ready state change" of 3 every time it receives data. Can you point me to the relevant part of the WHATWG spec?
It appears WebKit is not setting the responseText variable until the request has completed. Is there a way to access this text prior to state 4?
From code inspection, I see that XMLHttpRequest updates responseText every time it receives data.
Perhaps you're seeing the results of slightly different networking implementations. For example, you might need to send data in larger chunks in order to convince the networking layer to flush its data up to the application level.
On 6/26/07, Christopher Allen <ChristopherA@iphonewebdev.com> wrote:
We've written up a simple example using perl cgi and js/xhr to demonstrate this problem. In Firefox, the expected behavior appears. Every second, a new line appears counting up from 1 to 10. In Safari, there's a 10 second delay with nothing, and then 10 lines appear counting up from 1 to 10 all at once, just as the readyState goes from 3 to 4. We never see readyState going from 3 to 3 as we do with Firefox.
URL demonstrating this behavior: http://www.synchroedit.com/pt/
URL with source code for this test: http://www.synchroedit.com/pt/perl-test.tar.gz
The perl cgi script has autoflushing enabled, which means the buffer is flushed for every newline. In Firefox, the responseText is updated whenever the perl side sends and flushes data, while Safari's responseText stays empty until the perl cgi closes the connection and the request enters readyState 4.
On 6/26/07, Geoffrey Garen <ggaren@apple.com> wrote:
Is this capability supposed to be to in Safari, thus our test above demonstrates a bug?
At first glance, this seems like a bug. I think the best way to proceed is to get your data into a bug report @ bugs.webkit.org.
On 6/27/07, Kalle Alm <kalle.alm@gmail.com> wrote:
Sorry for spamming the list, but thought I'd add that I've filed this as a bug here: http://bugs.webkit.org/show_bug.cgi?id=14431
On 6/27/07, Kalle Alm alm@gmail.com> wrote:
Filed Apple bug as well -- the URL looks weird, but maybe it works. Problem ID is 5298296; URL which may or may not work is: https://bugreport.apple.com/cgi-bin/WebObjects/RadarWeb.woa/1/wo/P1gr2UkN22B...
participants (4)
-
Christopher Allen
-
Geoffrey Garen
-
Kalle Alm
-
Maciej Stachowiak