Improved validation

With the new filter syntax I introduced about a month ago, I did also see an increase in error reports due to issues with the filter syntax. Most of the time these errors where caused by user input (sorry to say this), for example because in the filter it was tested if a string is bigger than an integer or because the filter only listed a variable name but without a condition.

I have now improved the validation, so that these errors will also be reported while you type your script. To be able to do this I had to implement a “quick attribute scan” feature that will check your source data for the types of the attributes. Since this involves opening your geo data files I decided to schedule this check a bit less frequent than the normal validation, else the performance would probably be hurt too much. So it could take a few seconds before all errors related to the filter appear.

Hopefully these changes help you to write the correct filter syntax more easily. And hopefully it also reduces the amount of errors being reported.

New scenProc features

From tomorrow the scenProc 2.0 development release will have some new features included. Here is an overview of the main changes.

Polygon autogen library objects

The main change is that autogen library objects are now created from polygon instead of point features. This is in line with how Annotator creates these objects as well. The benefit of this is that you can now also create autogen library objects from building footprint data more easily. With the new CreateAGNLibObject step you can create the autogen library objects from polygons. The old way to create them from point is still available with the CreateAGNLibObj step that will be deprecated in the near future.

PointToPolygon step

Since library objects are made from polygons now, there are situations where you need to turn point features into polygon features. For this a new step has been added PointToPolygon. This step will create polygons with the length and width that you specify.

The existing ExtrudeLine step has been renamed to LineToPolygon step as well, so that the steps to convert between feature types have more logical names. The existing steps to create points from lines will be renamed to LineToPoint in the future as well to make everything more logical.

Updated tooltips

A third change is that I have changed the way that the help tooltips about the steps are generated. Before the information for the help text, the argument types and the sample line were spread in different places of my code. They are now in one place and that should make sure they stay in sync more (some of the help text was a bit outdated). So this new change should make the tooltips even more clear and accurate. But since I made a lot of changes now to the code, please let me know if some mistakes have slipped in.

 

Is it easy to waste memory?

mugI guess you already know the answer to the question posted in the title of this post. I’m afraid the answer was yes. This week two users (thanks to both of you) reported to me that scenProc is quite hungry for memory, especially when processing big files. Even 16 GB of RAM would easily be filled in certain cases.

So I started to check where all this memory went and the reason was quite logical. ModelConverterX and scenProc share a lot of code, they use the same libraries for many of the processing functions.  But the data structures designed initially for ModelConverterX were not always optimal for scenProc as well. In ModelConverterX for each vertex information like the position, normal, texture coordinate and bone weight are stored. scenProc doesn’t need all this information, only the position is enough. But it was using the same class to represent the vertices. So that gave a huge waste of memory.

I have now optimized this and also changed a few other things how scenProc stores the lines and polygons. And the good news is that this results in a much better memory usage. A reduction of 50% to 70% is possible for most scripts. So I’m sure you’ll notice the difference. And processing times also slightly improved in most cases because the information is stored more efficiently.

So grab the development release tomorrow and have a look yourself. Oh, and please don’t fill up the RAM that you freed with even more data. Because I’m sure some of you will go to the limit again 😉

scenProc updated filter syntax

The next development release of scenProc will contain the new filter syntax. Please have a good look at the updated manual, as it shows how the new syntax works and contains updated samples of all steps. Here are just the highlights of the changes:

  • Besides And conditions, you can now also do Or or Not conditions.
  • Different attribute types are now supported, before every attribute was seen as a string, but now you can also have integers or doubles.
  • You can do math within your filters, so for example sum two attributes or apply a math function on an attribute.
  • scenProc will automatically try to update the old filter syntax and in 95% of the cases that should work fine.
  • One of the biggest changes to be aware of when writing the filter is that if you compare to a text, you need to put quotes around it

I hope you all get used quickly to the new syntax, because it is a lot more powerful. And if there are any questions just post them in the forum.

Image2016-03-25 2133.56.161

New filter syntax almost there…

instruction-manualJust a early warning, the scenProc update with the new filter syntax is almost ready for release. I have just started to update the manual for all the changes, so hopefully in a few days I can put this feature in the development release.

Why this warning? This update will change how the filter syntax works, which might impact your existing scripts. There is logic in the tool to translate from the old to the new syntax and in 98% of the cases that should work. But if you want to be really sure, you better keep using scenProc 1.1 for your production work.

If however you want to use the new features (Or selections in the filter, the ability to do math with attributes in the filter, the ability to use Math functions in your filter) you should have a look at the new filter syntax once it’s released.

And now back to writing the manual…..

More exporting formats

I have added a new step to scenProc that provide more exporting possiblities. Besides the FS specific BGL and AGN export, before you could only export to images and Shapefiles. I have now added a new step that also allows to export to other GIS vector formats.

For loading vector data scenProc uses the OGR library for a while already, I have now added an ExportOGR step that can also write using this powerful library. This means you can not only write to Shapefile now, but also to other formats like GML or KML (and many more).

Please check the manual (which was also updated) for the specifics of this new step. Since the OGR library needs a little more information there are more attributes to specify than in the old ExportSHP step.

Let me know if there are any issues with this new step. In a while I plan to make the old ExportSHP step deprecated and then this step will be the main way to export vector data. But until it has been tested more, I’ll keep both for a while.

scenProc development release goes 2.0

The scenProc development release has now been updated to version 2.0. One of the first changes I have pushed into the development release is an update of the GDAL libraries to a more recent version. So if you grab the development release 2.0 for the first time, please do a full download and not an update download.

For the rest there should only be minor changes compared to the 1.1 version. The big changes to the filter syntax and to the way library objects are generated are still in my development branches and will make it to the development release once they have been tested more.

scenProc 1.1 stable released!

I have just released the stable version of scenProc 1.1. Compared to the old stable release 1.0 the changes made recently have been included. These are not really big changes, but since scenProc 2.0 will have some bigger changes coming up soon, I have decided to release a new stable release that people can fall back to.

You can find all the download links at the scenProc page. And in a few days I’ll update the development release to version 2.0 and start to roll out the new features I have been working on recently.

Build server update

I have updated to a different build server today. I won’t bother you with all the technical details, but this means that I use different software now to create the development releases of my tools automatically at night. For you as a user nothing changes, all download links stay they same. It’s only the build software in the background that I have updated. From my tests it seemed the migration went fine, but if you notice anything weird with the development release in the next day’s let me know. 

ImportOGR and automatic projection detection

I have now modified the ImportOGR step of scenProc. For the projection you can now specify a value of AUTODETECT to tell scenProc to try to detect the projection used by itself. This works for most vector data formats that GDAL/OGR supports, as long as the projection data is specified correctly in the vector data. For example for shapefiles there is often a PRJ file that specifies the projection of the data.

If the automatic detection fails you can still specify it manually like it was done before. And the NOREPROJ value to specify the projection is already WGS84 geodetic coordinates is also still supported of course.