[Webkit-unassigned] [Bug 237474] New: Safari ITP deleting same site cookies when Service Worker in place

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=237474

            Bug ID: 237474
           Summary: Safari ITP deleting same site cookies when Service
                    Worker in place
           Product: WebKit
           Version: Safari 15
          Hardware: iPhone / iPad
                OS: iOS 15
            Status: NEW
          Severity: Major
          Priority: P2
         Component: Service Workers
          Assignee: webkit-unassigned at lists.webkit.org
          Reporter: erik.witt at baqend.com

Created attachment 453843

  --> https://bugs.webkit.org/attachment.cgi?id=453843&action=review

Number of reassigned users depending on how long they have not visited the website.

Hi WebKit team!

I have to report a bug that I am afraid there is no simple way to reproduce but which has an immediate impact on lots of pages.

The issue seems to be that Safari in all observed versions (from 14.1 to 15.3) sometimes deletes server set same-origin cookies after 7 days of inactivity when a Service Worker is used on the page.

We suspect that this is linked to ITP which makes it virtually impossible to create an easily reproducible set up for you to try.

Instead, let me explain how we are coming to this conclusion and what our test setup looks like:

* We were running an AB test on a domain example.com in production. The group was chosen randomly on the server side and sent to the browser as a same-origin, secure but not HTTP-only cookie. The group stays the same for the user as long as he sends back the cookie on every navigation.
* In the client, we installed a Service Worker if the user was assigned to group A and did nothing when the user was assigned to group B.
* The Service Worker itself is rather simple. It caches static assets and stores some metadata in IndexedDB but it does not change the navigate request with the cookie at all.

We used a real user monitoring to compare the performance of both groups. In this data we can see the issue:

While it happens sometimes that the cookie gets lost and a user is reassigned in the AB test, this should happen about the same number of times in both groups.

What we saw is that much more users were reassigned a new group when they were in group A (with the Service Worker) than in group B.

Looking at those reassignments in more detail. This is what we found:

[1] The attached graph shows the number of reassigned users depending on how long they have not visited the website.

What you can clearly see is that when a user returns within 6 days to the website, the probability of him being reassigned is the same in both groups. 
If the user returns after 7 days or later, the probability is much higher to be reassigned if the user is in group A.

After a one-month AB test with more than 200k users per group, this effect resulted in group A having 3.5% fewer users (and even fewer returning users) which skews the conversion rate comparison we wanted to do in that AB test immensely.

There are four more insights we got from the data
* This discrepancy between the groups only happened for browser versions that had the Service Worker active (we had not installed it in Safari 15.1 and those users were fine)
* The discrepancy between the groups also did not happen for users from certain ASN (like akamai or cloudflare). It turned out to be users that had icloud private relay enabled.
* For a different AB test on a different domain we know that the same issue is not observed if the cookie used for the split test is marked as HTTP-Only.
* The issue seems to happen on mobile only

Probably needless to say but we have seen this issue only in Safari, no other browser.

For us it looks like ITP deletes too much data from the client if a service worker and indexeddb is used in the client.

If you have any other idea that explains the data we are seeing, or if you have a way to reproduce or debug an issue like this, it would be awesome to get your feedback. We are very open to helping in the investigation here!

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-unassigned/attachments/20220304/c4d563fc/attachment.htm>