Re: [webkit-dev] New iOS versions sending bogus User-Agent build data
On Thu, Apr 26, 2018 at 7:15 AM, Ricky Young <ricky@gmx.com> wrote:
I guess that soon it won't be possible to determine iOS version distribution using User-Agent sniffing. Right? Is there an alternative for the browser to tell the iOS version?
Hi, The very point of no longer sending the verison in the user agent is to make it difficult for websites to determine the operating system version. We already have to deal with enough serious web compatibility problems caused by the rest of the information in the user agent. I'm pleased that the version number is now stable -- that's a huge improvement -- but I won't be happy until Safari and other major browsers start sending fully-randomized UAs, or else a stub or blank UA ("Mozilla/5.0"). Michael
I'm running into this issue right now with the locked UA. Here are the challenges: 1. ios 11.3 safari bug broke <img src=mp4> despite advertising `accept: video/*` in the request. (see bug 185029). I now need to use UA detection to patch versions of a response based on existing bugs. In short, we as an industry, depend on UA parsing to work around bugs. This happened last year when the jp2k support increased to 10bit depth and the decoder was inflating the 8bit colour pallet to 10bit. We needed to use UA parsing to determine if it was safe to send jp2k images to a device. I have a laundry list of other examples where the server might actually know more than the client to prevent a) a broken user experience and b) prevent cellular data waste. 2. Javascript based feature detection is inefficient and slows down safari' preloader. The preloader is one of the best performance enhancements to come to the modern browser. Using javscript for feature detection is a step backward for the preloader. Webkit does not provide enough declarations of its feature set in HTTP headers so we resort to UA device characteristics. If the features are available with javascript inquiry, then how is that any different than exposing these features at request time? In short, we are stuck. blocking the UA like this will degrading the user experience because it will force us to either not take advantage of cell-data/cpu/battery improvements (like sending mp4s instead of animated gifs) or it will force us to use javascript and likewise degrade the user experience with slower web experiences. How do we navigate this? /colin On 26 April 2018 at 11:22, Michael Catanzaro <mcatanzaro@igalia.com> wrote:
On Thu, Apr 26, 2018 at 7:15 AM, Ricky Young <ricky@gmx.com> wrote:
I guess that soon it won't be possible to determine iOS version distribution using User-Agent sniffing. Right? Is there an alternative for the browser to tell the iOS version?
Hi,
The very point of no longer sending the verison in the user agent is to make it difficult for websites to determine the operating system version.
We already have to deal with enough serious web compatibility problems caused by the rest of the information in the user agent. I'm pleased that the version number is now stable -- that's a huge improvement -- but I won't be happy until Safari and other major browsers start sending fully-randomized UAs, or else a stub or blank UA ("Mozilla/5.0").
Michael
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
On Thu, Apr 26, 2018 at 10:57 AM, Colin Bendell | +1.613.914.3387 <colin@bendell.ca> wrote:
How do we navigate this?
By fixing the WebKit bug, of course. And in the meantime you can work around it on the server side by not using <img src=mp4>, right? Michael
On Thu, Apr 26, 2018 at 11:13 AM, Michael Catanzaro <mcatanzaro@igalia.com> wrote:
By fixing the WebKit bug, of course. And in the meantime you can work around it on the server side by not using <img src=mp4>, right?
Consider the other perspective on this problem. If other servers look at the WebKit version in the UA to determine if WebKit supports img src=mp4, other WebKit ports that don't support this are going to be out of luck and get broken pages. I know that's not what you're doing -- you're looking at iOS version instead, and only doing it to work around a specific bug, which is much better -- but the problem of websites sending bad content based on bad user agent parsing is so severe that we don't have many good options, here. :/ Michael
26.04.2018, 19:16, "Michael Catanzaro" <mcatanzaro@igalia.com>:
On Thu, Apr 26, 2018 at 11:13 AM, Michael Catanzaro <mcatanzaro@igalia.com> wrote:
By fixing the WebKit bug, of course. And in the meantime you can work around it on the server side by not using <img src=mp4>, right?
Consider the other perspective on this problem. If other servers look at the WebKit version in the UA to determine if WebKit supports img src=mp4, other WebKit ports that don't support this are going to be out of luck and get broken pages. I know that's not what you're doing -- you're looking at iOS version instead, and only doing it to work around a specific bug, which is much better -- but the problem of websites sending bad content based on bad user agent parsing is so severe that we don't have many good options, here. :/
Not to mention those evil people who reject page loading for user agents they don't (want to) support -- Regards, Konstantin
On 26 April 2018 at 12:23, Konstantin Tokarev <annulen@yandex.ru> wrote:
26.04.2018, 19:16, "Michael Catanzaro" <mcatanzaro@igalia.com>:
On Thu, Apr 26, 2018 at 11:13 AM, Michael Catanzaro <mcatanzaro@igalia.com> wrote:
By fixing the WebKit bug, of course. And in the meantime you can work around it on the server side by not using <img src=mp4>, right?
Today, we can work around it by ignoring the Accept field. However, tomorrow once the bug is fixed we will be in the paradox of not knowing which versions of Safari are telling the truth.
Consider the other perspective on this problem. If other servers look at the WebKit version in the UA to determine if WebKit supports img src=mp4, other WebKit ports that don't support this are going to be out of luck and get broken pages. I know that's not what you're doing -- you're looking at iOS version instead, and only doing it to work around a specific bug, which is much better -- but the problem of websites sending bad content based on bad user agent parsing is so severe that we don't have many good options, here. :/
Not to mention those evil people who reject page loading for user agents they don't (want to) support
How does locking the UA solve the misbehaving parsers in the wild? They will still misbehave and break the user experience. However, those of us that are trying to optimize the user experience by working around bugs for specific versions are now handicapped and punished. The reality is that bugs exist. There is a give and take here where the content negotiation is sometimes smarter on the client, and sometimes smarter on the server. In the case of media, the server is usually smarter and must be because of the long tail of adoption. Ideally everyone on the planet will adopt to ios V.Latest overnight, but sadly this will not be the case. Servers must account for the long tail of users on many different versions. Right now we are forced to decide to apply a solution that works only for v.latest, or apply a solution that works for v.lowest-common-denominator. In both scenarios we compromise the user experience. Ideally we wouldn't have to sacrifice some users and would be able to account for bugs in the wild. As I said before, if this detail is available in javascript, then why hide it on the wire? Perhaps it's time for a User-Agent2 with precise feature declarations? Even still a version number is needed because, well, bugs. While I understand the fingerprinting concern, I'm not particularly convinced that UA obfuscation is the best solution for users and the user experience. Without opening up the religious debate of fingerprinting with the UA, how can provide enough detail (version numbers) so that service providers can work toward the greater good of improving the user experience? Is there room for compromise? /colin
26.04.2018, 20:09, "Colin Bendell | +1.613.914.3387" <colin@bendell.ca>:
On 26 April 2018 at 12:23, Konstantin Tokarev <annulen@yandex.ru> wrote:
Not to mention those evil people who reject page loading for user agents they don't (want to) support
How does locking the UA solve the misbehaving parsers in the wild?
There will be less information in UA and therefore less reasons for developers to do this, instead of using direct feature testing
They will still misbehave and break the user experience. However, those of us that are trying to optimize the user experience by working around bugs for specific versions are now handicapped and punished.
While parsing UA string handicaps and punishes users of alternative user agents. -- Regards, Konstantin
On 26 April 2018 at 13:18, Konstantin Tokarev <annulen@yandex.ru> wrote:
26.04.2018, 20:09, "Colin Bendell | +1.613.914.3387" <colin@bendell.ca>:
On 26 April 2018 at 12:23, Konstantin Tokarev <annulen@yandex.ru> wrote:
Not to mention those evil people who reject page loading for user agents they don't (want to) support
How does locking the UA solve the misbehaving parsers in the wild?
There will be less information in UA and therefore less reasons for developers to do this, instead of using direct feature testing
They will still misbehave and break the user experience. However, those of us that are trying to optimize the user experience by working around bugs for specific versions are now handicapped and punished.
While parsing UA string handicaps and punishes users of alternative user agents.
Can you give me an example where UA parsing is punishing users of alternative user agents? Is this a theoretical problem, or a widespread problem? I'm not asking to be divisive, but because I know for a fact that UA parsing is improving the user experience. I can give you dozens of examples where we must resort to UA parsing for the betterment of the user (for all flavors of UAs). Do you have any alternatives other than "more javascript"? Again I ask, is there room for compromise where we can expose the version details in the UA (or some alternative) so that we ensure a consistent and optimized user experience? /colin
It's worth noting that https://bugs.webkit.org/show_bug.cgi?id=182629 added back the OS version to the UA string, at least on trunk (the reasons given there, in https://bugs.webkit.org/show_bug.cgi?id=182629#c6, sound exactly like those mentioned by Colin in this thread). On Thu, Apr 26, 2018 at 1:49 PM Colin Bendell | +1.613.914.3387 < colin@bendell.ca> wrote:
On 26 April 2018 at 13:18, Konstantin Tokarev <annulen@yandex.ru> wrote:
26.04.2018, 20:09, "Colin Bendell | +1.613.914.3387 <(613)%20914-3387>" <colin@bendell.ca>:
On 26 April 2018 at 12:23, Konstantin Tokarev <annulen@yandex.ru> wrote:
Not to mention those evil people who reject page loading for user agents they don't (want to) support
How does locking the UA solve the misbehaving parsers in the wild?
There will be less information in UA and therefore less reasons for developers to do this, instead of using direct feature testing
They will still misbehave and break the user experience. However, those of us that are trying to optimize the user experience by working around bugs for specific versions are now handicapped and punished.
While parsing UA string handicaps and punishes users of alternative user agents.
Can you give me an example where UA parsing is punishing users of alternative user agents? Is this a theoretical problem, or a widespread problem? I'm not asking to be divisive, but because I know for a fact that UA parsing is improving the user experience. I can give you dozens of examples where we must resort to UA parsing for the betterment of the user (for all flavors of UAs).
Do you have any alternatives other than "more javascript"?
Again I ask, is there room for compromise where we can expose the version details in the UA (or some alternative) so that we ensure a consistent and optimized user experience?
/colin _______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
On Thu, Apr 26, 2018 at 12:48 PM, Colin Bendell | +1.613.914.3387 <colin@bendell.ca> wrote:
Can you give me an example where UA parsing is punishing users of alternative user agents? Is this a theoretical problem, or a widespread problem? I'm not asking to be divisive, but because I know for a fact that UA parsing is improving the user experience. I can give you dozens of examples where we must resort to UA parsing for the betterment of the user (for all flavors of UAs).
I would say it's the most serious web compatibility problem that exists today. Our users complain and blame us when important websites are broken in WebKit because of it. I have personally wasted days [1] of [2] development [3] effort [4] playing with WebKit's user agent quirks to get important websites to work properly. You can look at the history of our quirks list for non-Safari ports [5] to get an idea of the trouble we've had to deal with in recent years. That doesn't count cases like [6] where I suspect user agent trouble, but just don't care to investigate. As a web engine developer, it's hard to understate how frustrating this is, especially in cases like [7] where we had tons of users complaining and the quirks were particularly difficult to write. It sometimes feels like website developers are our enemy, just out to break our web engine. Sometimes that's even true, e.g. when websites intentionally decide to block access to unrecognized browsers, or scare our users with claims that a browser is "unsupported" even though it works fine with a UA quirk. This is not a healthy situation for the web. If everyone was careful and responsible with how they use the user agent, it wouldn't be a problem, but at this point it's long been spoiled for everyone. I'm sorry that this fallout affects developers like you who are just trying to work around a bug. :( On Thu, Apr 26, 2018 at 12:48 PM, Colin Bendell | +1.613.914.3387 <colin@bendell.ca> wrote:
Again I ask, is there room for compromise where we can expose the version details in the UA (or some alternative) so that we ensure a consistent and optimized user experience?
I don't know. I wish there was, but I don't think so. If you have any suggestions, that'd be great, but I think it's going to be extremely difficult or impossible to solve this problem in a way that makes everyone happy. I don't think I'll be happy until major browsers send the same, standardized user agent as other major browsers. (Or send fully-randomized user agents, but that's probably an impossible dream.) Freezing just the version numbers is not good enough, but it's a step in the right direction, and I really appreciate it. [1] https://trac.webkit.org/changeset/206519/webkit [2] https://trac.webkit.org/changeset/216139/webkit [3] https://trac.webkit.org/changeset/216343/webkit [4] https://trac.webkit.org/changeset/217203/webkit [5] https://trac.webkit.org/log/webkit/trunk/Source/WebCore/platform/UserAgentQu... [6] https://bugs.webkit.org/show_bug.cgi?id=180995 [7] https://bugs.webkit.org/show_bug.cgi?id=171770#c4
On 26 April 2018 at 14:25, Ali Juma <ajuma@chromium.org> wrote:
It's worth noting that https://bugs.webkit.org/show_bug.cgi?id=182629 added back the OS version to the UA string, at least on trunk (the reasons given there, in https://bugs.webkit.org/show_bug.cgi?id=182629#c6, sound exactly like those mentioned by Colin in this thread).
Thanks for the catch. I had missed this commit. This is helpful. On 26 April 2018 at 15:40, Michael Catanzaro <mcatanzaro@igalia.com> wrote:
I would say it's the most serious web compatibility problem that exists today. Our users complain and blame us when important websites are broken in WebKit because of it. I have personally wasted days [1] of [2] development [3] effort [4] playing with WebKit's user agent quirks to get important websites to work properly.
Yea, I get that. The current state of the UA is terrible. It's really an archeological map of browser evolution. It's like going on a first-date and the first thing you talk about is your family tree, and how great-grandpa-andreesen who is all but estranged from the family. It is a tad embarrassing :) On 26 April 2018 at 15:40, Michael Catanzaro <mcatanzaro@igalia.com> wrote:
On Thu, Apr 26, 2018 at 12:48 PM, Colin Bendell <colin@bendell.ca> wrote:
Again I ask, is there room for compromise where we can expose the version details in the UA (or some alternative) so that we ensure a consistent and optimized user experience?
I don't know. I wish there was, but I don't think so. If you have any suggestions, that'd be great, but I think it's going to be extremely difficult or impossible to solve this problem in a way that makes everyone happy.
I was only partially joking about the need for a UA2 spec. Perhaps it is time to bring this to one of the w3c working groups. I'll take it off this thread though :) /colin
Hi Colin,
On Apr 26, 2018, at 8:57 AM, Colin Bendell | +1.613.914.3387 <colin@bendell.ca> wrote:
[…] images to a device. I have a laundry list of other examples where the server might actually know more than the client to prevent a) a broken user experience and b) prevent cellular data waste.
Would you mind documenting those items in a Bugzilla report? We are interested in specific use cases that suffer with a frozen user agent string, and what (apparently bad) techniques you need to use to work around it. Thanks, -Brent
I can totally understand your intentions, it feels like that in 2018 UA should not be a relevant piece of information, and especially not be a factor in the decision if a server should serve a website to a certain request. But UA is how the Internet currently works. Correct me if I'm wrong, but it feels like you are trying to change the Internet. Are you trying to get rid of it for good? I understand that these kind of drastic, revolutionary, changes might take casualties, I feel like the industry was not given a chance to adopt to this change. Sent: Thursday, April 26, 2018 at 6:22 PM From: "Michael Catanzaro" <mcatanzaro@igalia.com> To: "Ricky Young" <ricky@gmx.com> Cc: "Anne van Kesteren" <annevk@annevk.nl>, "webkit-dev@lists.webkit.org" <webkit-dev@lists.webkit.org> Subject: Re: [webkit-dev] New iOS versions sending bogus User-Agent build data On Thu, Apr 26, 2018 at 7:15 AM, Ricky Young <ricky@gmx.com> wrote:
I guess that soon it won't be possible to determine iOS version distribution using User-Agent sniffing. Right? Is there an alternative for the browser to tell the iOS version?
Hi, The very point of no longer sending the verison in the user agent is to make it difficult for websites to determine the operating system version. We already have to deal with enough serious web compatibility problems caused by the rest of the information in the user agent. I'm pleased that the version number is now stable -- that's a huge improvement -- but I won't be happy until Safari and other major browsers start sending fully-randomized UAs, or else a stub or blank UA ("Mozilla/5.0"). Michael
participants (7)
-
Ali Juma
-
Brent Fulgham
-
Colin Bendell
-
Colin Bendell | +1.613.914.3387
-
Konstantin Tokarev
-
Michael Catanzaro
-
Ricky Young