Is it easy to waste memory?

mugI guess you already know the answer to the question posted in the title of this post. I’m afraid the answer was yes. This week two users (thanks to both of you) reported to me that scenProc is quite hungry for memory, especially when processing big files. Even 16 GB of RAM would easily be filled in certain cases.

So I started to check where all this memory went and the reason was quite logical. ModelConverterX and scenProc share a lot of code, they use the same libraries for many of the processing functions.  But the data structures designed initially for ModelConverterX were not always optimal for scenProc as well. In ModelConverterX for each vertex information like the position, normal, texture coordinate and bone weight are stored. scenProc doesn’t need all this information, only the position is enough. But it was using the same class to represent the vertices. So that gave a huge waste of memory.

I have now optimized this and also changed a few other things how scenProc stores the lines and polygons. And the good news is that this results in a much better memory usage. A reduction of 50% to 70% is possible for most scripts. So I’m sure you’ll notice the difference. And processing times also slightly improved in most cases because the information is stored more efficiently.

So grab the development release tomorrow and have a look yourself. Oh, and please don’t fill up the RAM that you freed with even more data. Because I’m sure some of you will go to the limit again 😉

4 thoughts on “Is it easy to waste memory?

  1. Kelvin says:

    Awesome. I have 64GB Ram and it didn’t take long for that to be chewed up, although I process a lot of data so thought it was normal. Great work.

  2. samlian says:

    I have 24gb of ram, maximum supported from my MB, with osm over 7,3gb it’s out of memory, i think to split osm date, is there another way? thank

    • arno says:

      Is the 7.3 GB the amount of memory scenProc uses when it gives an out of memory error? Or do you mean that you can load a 7.3 GB OSM file into scenProc? A 1 GB OSM file doesn’t necessary translate to 1 GB of memory usage.

      Arno

      • samlian says:

        scenproc give me out of memory message when it uses all memory (24gb) i loaded one 7.3gb osm file into scenproc,

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.