[Webkit-unassigned] [Bug 97643] Flakiness Dashboard server OOMs when the results.json gets too large

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=97643


Ojan Vafai <ojan at chromium.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|Flakiness Dashboard does    |Flakiness Dashboard server
                   |not accept results from EFL |OOMs when the results.json
                   |WK2 Bot                     |gets too large
                 CC|                            |dpranke at chromium.org




--- Comment #1 from Ojan Vafai <ojan at chromium.org>  2012-09-26 11:55:35 PST ---
I've fixed the glitch.

I deleted the "show all runs" data for this bot. Deleting the data for the bot isn't a big deal since we only keep the last 500 runs anyways, it's just a temporary data loss.

This is a long-standing bug when the accumulated data in the results.json gets too large), the python server runs out of memory trying to parse it. We delete runs older than 500 and we delete entries that have only passed or been skipped in the past 500 runs. So the results.json is usually self-pruning and we don't hit this. But if to too many different tests fail in the past 500 runs, we get stuck here.

There are a couple of proposed solutions, but noone has had the time to implement them:
1. Move over to using AppEngine Backend servers: https://developers.google.com/appengine/docs/python/backends/overview
2. Use a TaskQueue to do the JSON merging https://developers.google.com/appengine/docs/python/taskqueue/overview
3. Chunk the json we store every 100 runs and make the dashboard UI load 100 run chunks at a time. This would solve both the memory problem and would have the benefit of making it so we don't have to delete data older than 500 runs.

Due to http://code.google.com/p/googleappengine/issues/detail?id=7973 we can't get the error logs to show us which builders are having this problem. :(

-- 
Configure bugmail: https://bugs.webkit.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.