<div dir="ltr"><div>I don't think this is really 32bit vs 64bit platform issue. The vast majority of 64bit systems our users have (that is iOS devices) can't use memory buffers sized anywhere near the 32bit limit even in theory. Also when using Vector auto-grow capabilities (which is really the point of using a vector instead of just allocating a buffer) you need way more memory than the actual data size. Growing a Vector<uint8_t> beyond 4GB has peak allocation of 9GB.</div><div><br></div>Are there any examples of Vectors in the current code base where we would usefully fix an actual problem even in medium term by switching to 64bit storage sizes? I don't think they exists. Cases like file uploads should stream the data or use some mapped-file backed buffer type that is not a Vector.<div><br></div><div>With this in mind I think the right direction is to make the Vector API match the implementation and just start using unsigned indexes everywhere.</div><div><br></div><div><br></div><div> antti</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Nov 21, 2014 at 2:59 AM, Maciej Stachowiak <span dir="ltr"><<a href="mailto:mjs@apple.com" target="_blank">mjs@apple.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div><div class="h5"><br><div><blockquote type="cite"><div>On Nov 20, 2014, at 4:51 PM, Maciej Stachowiak <<a href="mailto:mjs@apple.com" target="_blank">mjs@apple.com</a>> wrote:</div><br><div><div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><blockquote type="cite"><div><br>On Nov 20, 2014, at 9:26 AM, Alexey Proskuryakov <<a href="mailto:ap@webkit.org" target="_blank">ap@webkit.org</a>> wrote:</div><br><div><div style="word-wrap:break-word"><br><div><div>19 нояб. 2014 г., в 14:58, Alexey Proskuryakov <<a href="mailto:ap@webkit.org" target="_blank">ap@webkit.org</a>> написал(а):</div><br><blockquote type="cite"><div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">These and related uses are all over the place - see also Vectors in FormDataBuilder, data returned from FrameLoader::loadResourceSynchronously, plug-in code that loads from network, SharedBuffer etc.</div></blockquote><br></div><div>Another way to say this is: we need to deal with large data arrays throughout loading code. We do not really need full size vectors in most other code - it's sort of OK for HTML parser or for image decoder to fail catastrophically when there is too much data fed to it.</div><div><br></div><div>This is somewhat questionable design, but if we are going to stick to it, then magnitude checks should be centralized, not sprinkled throughout the code. We should not make this check explicitly when feeding a network resource to the parser, for example.</div><div><br></div><div>A 64-bit API for Vector solves this nearly flawlessly. We do not perform the checks manually every time we use a Vector, Vector does them for us.</div><div><br></div><div>Other options are:</div><div><br></div><div>- uint64_t everywhere. This way, we'll solve practical problems with large resources once and for all. Also, this may prove to be necessary to solve even YouTube/Google Drive uploads, I do not know that yet.</div><div><br></div><div>- size_t everywhere. Same effect on 64-bit platforms, while 32-bit ones will still be limited. I like this option, because it won't make us pay the memory and performance cost on old crippled 32-bit machines, which are unlikely to be used for manipulating huge volumes of data anyway.</div></div></div></blockquote><br></div><div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">We probably want YouTube upload of large files to work on 32-bit machines. Though presumably if you want to upload a file larger than the virtual address space, you need to represent it in some way other than a Vector.</div></div></blockquote><br></div></div></div><div>Thinking about this more - I think file sizes should probably be an off_t, not a size_t, so on 32-bit platforms some care must still be taken in the conversion between file sizes and vector sizes.</div><div><br></div><div>In general, I like the idea of Vector having a size_t API and a choice of smaller or larger internal size. We might also want to change other containers with size-related interfaces to match.</div><div><br></div><div>Regards,</div><div>Maciej</div><div><br></div><br></div><br>_______________________________________________<br>
webkit-dev mailing list<br>
<a href="mailto:webkit-dev@lists.webkit.org">webkit-dev@lists.webkit.org</a><br>
<a href="https://lists.webkit.org/mailman/listinfo/webkit-dev" target="_blank">https://lists.webkit.org/mailman/listinfo/webkit-dev</a><br>
<br></blockquote></div><br></div>