scenProc updated filter syntax

The next development release of scenProc will contain the new filter syntax. Please have a good look at the updated manual, as it shows how the new syntax works and contains updated samples of all steps. Here are just the highlights of the changes:

  • Besides And conditions, you can now also do Or or Not conditions.
  • Different attribute types are now supported, before every attribute was seen as a string, but now you can also have integers or doubles.
  • You can do math within your filters, so for example sum two attributes or apply a math function on an attribute.
  • scenProc will automatically try to update the old filter syntax and in 95% of the cases that should work fine.
  • One of the biggest changes to be aware of when writing the filter is that if you compare to a text, you need to put quotes around it

I hope you all get used quickly to the new syntax, because it is a lot more powerful. And if there are any questions just post them in the forum.

Image2016-03-25 2133.56.161

New filter syntax almost there…

instruction-manualJust a early warning, the scenProc update with the new filter syntax is almost ready for release. I have just started to update the manual for all the changes, so hopefully in a few days I can put this feature in the development release.

Why this warning? This update will change how the filter syntax works, which might impact your existing scripts. There is logic in the tool to translate from the old to the new syntax and in 98% of the cases that should work. But if you want to be really sure, you better keep using scenProc 1.1 for your production work.

If however you want to use the new features (Or selections in the filter, the ability to do math with attributes in the filter, the ability to use Math functions in your filter) you should have a look at the new filter syntax once it’s released.

And now back to writing the manual…..

More exporting formats

I have added a new step to scenProc that provide more exporting possiblities. Besides the FS specific BGL and AGN export, before you could only export to images and Shapefiles. I have now added a new step that also allows to export to other GIS vector formats.

For loading vector data scenProc uses the OGR library for a while already, I have now added an ExportOGR step that can also write using this powerful library. This means you can not only write to Shapefile now, but also to other formats like GML or KML (and many more).

Please check the manual (which was also updated) for the specifics of this new step. Since the OGR library needs a little more information there are more attributes to specify than in the old ExportSHP step.

Let me know if there are any issues with this new step. In a while I plan to make the old ExportSHP step deprecated and then this step will be the main way to export vector data. But until it has been tested more, I’ll keep both for a while.

scenProc development release goes 2.0

The scenProc development release has now been updated to version 2.0. One of the first changes I have pushed into the development release is an update of the GDAL libraries to a more recent version. So if you grab the development release 2.0 for the first time, please do a full download and not an update download.

For the rest there should only be minor changes compared to the 1.1 version. The big changes to the filter syntax and to the way library objects are generated are still in my development branches and will make it to the development release once they have been tested more.

scenProc 1.1 stable released!

I have just released the stable version of scenProc 1.1. Compared to the old stable release 1.0 the changes made recently have been included. These are not really big changes, but since scenProc 2.0 will have some bigger changes coming up soon, I have decided to release a new stable release that people can fall back to.

You can find all the download links at the scenProc page. And in a few days I’ll update the development release to version 2.0 and start to roll out the new features I have been working on recently.

Build server update

I have updated to a different build server today. I won’t bother you with all the technical details, but this means that I use different software now to create the development releases of my tools automatically at night. For you as a user nothing changes, all download links stay they same. It’s only the build software in the background that I have updated. From my tests it seemed the migration went fine, but if you notice anything weird with the development release in the next day’s let me know. 

ImportOGR and automatic projection detection

I have now modified the ImportOGR step of scenProc. For the projection you can now specify a value of AUTODETECT to tell scenProc to try to detect the projection used by itself. This works for most vector data formats that GDAL/OGR supports, as long as the projection data is specified correctly in the vector data. For example for shapefiles there is often a PRJ file that specifies the projection of the data.

If the automatic detection fails you can still specify it manually like it was done before. And the NOREPROJ value to specify the projection is already WGS84 geodetic coordinates is also still supported of course.

ImportGDAL and projections

Until now the scenProc ImportGDAL step required that the imagery you are reading was in WGS84 projection already. I have now changed this and other projections are also supported. If the imagery is not in WGS84 it will be reprojected by scenProc internally. This should make it even easier to run the feature detection feature on the imagery you have, as many sources don’t provide the imagery in WGS84 projection. So it should save quite some pre-processing of your data.

scenProc will try to automatically detect the projection that is used in your imagery. In most cases that works fine, if this doesn’t work out you can manually provide the projection as an additional argument to the ImportGDAL step.

There is one little catch, the internal reprojection is done in memory. So if you are already short on memory for your scenProc script, it might be better to still reproject your imagery manually. But from the tests I have done until now this shouldn’t be needed too often.

scenProc big area tips (2)

Here is another tip when using scenProc to process big areas. If you are working with raster data this can mean that you have to load a big amount of files. But there is an easier way. scenProc uses the GDAL library that supports reading GeoTIFF, JPEG2000 and other common formats. But it also supports WMS.

WMS stands for Web Mapping Service and is a technology to distribute imagery over the internet. It is this kind of technology that gives you the images you see in Google Earth for example.

To allow scenProc to read the imagery all you have to do is make a XML configuration file that specifies the WMS service. Below you will find an example and all details can be found in the GDAL documentation. Then you use this XML file as filename in the ImportGDAL step. Then the images from the WMS can be used in the steps that use raster features. Be aware that grabbing all the data might take a while.

<GDAL_WMS>
 <Service name="WMS">
 <Version>1.1.1</Version>
 <ServerUrl>http://gis.apfo.usda.gov/arcgis/services/NAIP/Florida_2013_1m/ImageServer/WMSServer?service=WMS</ServerUrl>
 <Layers>0</Layers>
 <SRS>EPSG:4326</SRS>
 <ImageFormat>image/jpeg</ImageFormat>
 <Transparent>FALSE</Transparent>
 <BBoxOrder>xyXY</BBoxOrder>
 </Service>
 <DataWindow>
 <UpperLeftX>-87.71</UpperLeftX>
 <UpperLeftY>31.09</UpperLeftY>
 <LowerRightX>-79.96</LowerRightX>
 <LowerRightY>24.42</LowerRightY>
 <SizeX>855776</SizeX>
 <SizeY>826421</SizeY>
 </DataWindow>
 <BandsCount>3</BandsCount>
</GDAL_WMS>

scenProc big area tips (1)

When you are trying to process a big area in scenProc getting all vector data loaded can be hard. Either you will have some big shape files that take long to load or you need to do some pre-processing to slice up the data before. But did you know there is an easier way?

scenProc uses the OGR library to read the vector data. This library does not only support many formats, like shape files or GML, but it can also connect to geographical databases. So if you put all your geo data in such a database scenProc can read from it using the ImportOGR step. And only the data for the bounding coordinates that you specified will be read. Instead of a file name you need to specify a database connection string like this:

PG:"dbname='databasename' host='addr' port='5432' user='x' password='y'"

Of course initially you need to setup the database and put your data in it. The connection string above is for a PostgreSQL database with the PostGIS extension installed. There is an easy installer  to get all this installed quickly. And with ogr2ogr it’s also not so hard to put your data into the database.

So the next time you start a project where you will use scenProc to process a big areas consider setting up a geo database first to make your life a bit easier.