[webkit-qt] [Qt-interest] Multiple page rendering using webkit

[Tarandeep Singh]

The pages to be rendered are not on the disk and need to be fetched over
network.
Based on your replies, I am gonna try this-

Have multiple processes but each process should have couple of webpages, put
NAM in threads so as to improve network IO.

With multiple processes, I might run out of memory as I have seen memory
foot print of qt+webkit process keeps increasing as I keep rendering pages
over a long period of time, so I might need to do something about it  - like
restart the process after some time.

Thanks!


On Fri, Apr 15, 2011 at 1:55 PM, Jason H <scorp1us at yahoo.com> wrote:

> Yes, I'd probably use renderWidget() to get a pixmap and find the fastest
> way to
> get that to the master app. (local socket?)
> usability becomes an issue for anything more than a thumb nail. You'd need
> a way
> to map the mouse/keyboard events back through the socket and make the
> corresponding adjustments in the corresponding webkit process.
>
>
>
>
>
>
> ----- Original Message ----
> From: Arnold Krille <arnold at arnoldarts.de>
> To: qt-interest at trolltech.com
> Sent: Fri, April 15, 2011 4:46:41 PM
> Subject: Re: [Qt-interest] Multiple page rendering using webkit
>
> Hi,
>
> On Friday 15 April 2011 21:33:07 Tarandeep Singh wrote:
> > I need to render a large number of pages and obviously need do this as
> > efficiently as possible.
> > Since GUI rendering needs to be done on the main thread, so I can't
> create
> > multiple QWebPage threads.
> >
> > So I am thinking of these 2 solutions-
> >
> > 1) Create one process and have N QWebPage instances so that N pages can
> be
> > rendered simultaneously. Here assumption is QT+Webkit behind the scenes
> > create threads to do parallel IO etc.
> >
> > 2) Create N processes where each process handles 1 page request at a
> given
> > time.
> >
> > Can someone please advice which solution should work better or is there
> is
> > another way to achieve this.
> > I have 16 cores and 32G of RAM.
>
> I assume you have the to-be-rendered webpages on disk already as html and
> "only" want the rendered output in pdf/svg/png or whatever. (If not I think
> you can easily do with three to four webpages which in turns load the files
> from network and render and save to disk, net-io will be the slowest
> point.)
>
> As all gui-stuff has to run in the thread that instantiated QApplication,
> you
> can really only use one thread for rendering. But who says that you can't
> run
> multiple processes in parallel?
> One process, maybe even the master of your app, loads the html from disk
> into
> a queue which is in shared memory and then starts as many render-processes
> as
> you like. These render processes each take a pages html from the queue and
> render it and either put the rendered data back onto another shared queue
> or
> save directly to files on disk.
> If you want to leave room for other processes to run (like the desktop),
> you
> should run like min(1, idealThreadCount()/2) render processes.
>
> Hope that helps,
>
> Arnold
>
> _______________________________________________
> Qt-interest mailing list
> Qt-interest at qt.nokia.com
> http://lists.qt.nokia.com/mailman/listinfo/qt-interest
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-qt/attachments/20110415/d4f9c19f/attachment-0001.html>