K
Q

Geoserver issue with big raster

3/3/2022

General Cluster Information:

Kubernetes version: 1.19.13 Cloud being used: private Installation method: kubeadm init Host OS: Ubuntu 20.04.1 LTS CNI and version: Weave Net: 2.7.0 CRI and version: Docker: 19.3.13

I have some trouble running geoserver 2.20.2 (stable version) on my kubernetes cluster. I created the Docker image from scratch as follows:

FROM ubuntu:20.04
WORKDIR /root
RUN apt update; apt install wget -y; apt install unzip -y;  DEBIAN_FRONTEND=noninteractive TZ=Europe/Berlin apt install openjdk-11-jdk -y; apt install postgis postgresql-client-12 gdal-bin -y
RUN wget https://phoenixnap.dl.sourceforge.net/project/geoserver/GeoServer/2.20.2/geoserver-2.20.2-bin.zip; unzip geoserver-2.20.2-bin.zip; rm geoserver-2.20.2-bin.zip;
RUN wget https://iweb.dl.sourceforge.net/project/geoserver/GeoServer/2.17.5/extensions/geoserver-2.17.5-imagemosaic-jdbc-plugin.zip; unzip geoserver-2.17.5-imagemosaic-jdbc-plugin.zip; rm geoserver-2.17.5-imagemosaic-jdbc-plugin.zip; mv gt-imagemosaic-jdbc-23.5.jar webapps/geoserver/WEB-INF/lib/
RUN wget https://build.geoserver.org/geoserver/2.20.x/community-latest/geoserver-2.20-SNAPSHOT-pgraster-plugin.zip; unzip geoserver-2.20-SNAPSHOT-pgraster-plugin.zip; rm geoserver-2.20-SNAPSHOT-pgraster-plugin.zip; mv gs-pgraster-2.20-SNAPSHOT.jar webapps/geoserver/WEB-INF/lib/; 
CMD bash ./bin/startup.sh

My aim is to offer a wms for accessing raster data stored in a postgresql database (with postgis extension)

therefore, I first created the database and the postgis extension

createdb -U postgres -h databasehost  geotest
psql -U postgres -d geotest -h databasehost -c "CREATE EXTENSION postgis;"

Afterwards, I combined all raster files in a vrt-file

for file in *.tif 
do
        echo $file
        gdalwarp -t_srs EPSG:25832 -overwrite $file neu_${file}
        rm $file
        mv neu_$file $file
done
gdalbuildvrt composite.vrt *.tif

I created a folder tiles and execute gdal_retile to retile the created raster file

mkdir tiles
gdal_retile.py -co "WORLDFILE=YES" -r bilinear -ps 100 100 -of GTIFF -levels 4 -targetDir tiles composite.vrt

Subsequently, I created three files, as mentioned in the docs of geoserver

#connect.postgis.xml.inc
<connect>
  <!-- value DBCP or JNDI -->
  <dstype value="DBCP"/>
  <!--   <jndiReferenceName value=""/>  -->
  <username value="postgres" />
  <password value="password" />
  <jdbcUrl value="jdbc:postgresql://databasehost:5432/geotest" />
  <driverClassName value="org.postgresql.Driver"/>
  <maxActive value="10"/>
  <maxIdle value="0"/>
</connect>

#mapping.postgis.xml.inc
<!-- possible values: universal,postgis,db2,mysql,oracle -->
<spatialExtension name="postgis"/>
<mapping>
    <masterTable name="mosaic" >
      <coverageNameAttribute name="name"/>
      <maxXAttribute name="maxX"/>
      <maxYAttribute name="maxY"/>
      <minXAttribute name="minX"/>
      <minYAttribute name="minY"/>
      <resXAttribute name="resX"/>
      <resYAttribute name="resY"/>
      <tileTableNameAtribute  name="TileTable" />
      <spatialTableNameAtribute name="SpatialTable" />
    </masterTable>
    <tileTable>
      <blobAttributeName name="data" />
      <keyAttributeName name="location" />
    </tileTable>
    <spatialTable>
      <keyAttributeName name="location" />
      <geomAttributeName name="geom" />
      <tileMaxXAttribute name="maxX"/>
      <tileMaxYAttribute name="maxY"/>
      <tileMinXAttribute name="minX"/>
      <tileMinYAttribute name="minY"/>
    </spatialTable>
</mapping>

#dgm.postgis.xml
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE ImageMosaicJDBCConfig [
  <!ENTITY mapping PUBLIC "mapping"  "mapping.postgis.xml.inc">
  <!ENTITY connect PUBLIC "connect"  "connect.postgis.xml.inc">]>
<config version="1.0">
  <coverageName name="dgm"/>
  <coordsys name="EPSG:25832"/>
  <!-- interpolation 1 = nearest neighbour, 2 = bilinear, 3 = bicubic -->
  <scaleop  interpolation="1"/>
  <verify cardinality="false"/>
  &mapping;
  &connect;
</config>

Than I created a new folder for storing the postgresql files to create the necessary database tables and created the tables

mkdir sqlscripts
java -jar ~/webapps/geoserver/WEB-INF/lib/gt-imagemosaic-jdbc-23.5.jar ddl -config dgm.postgis.xml -spatialTNPrefix dgm -pyramids 4 -statementDelim ";" -srs 25832 -targetDir sqlscripts

psql -U postgres -d geotest -h databasehost -f sqlscripts/createmeta.sql
psql -U postgres -d geotest -h databasehost -f sqlscripts/add_dgm.sql

finally, I pushed the rastertiles to the database and copied the three configuration file mentioned above to the following directory: /root/data_dir/coverages/dgm

java -classpath /root/webapps/geoserver/WEB-INF/lib/gt-imagemosaic-jdbc-23.5.jar:/root/webapps/geoserver/WEB-INF/lib/postgresql-42.2.19.jar:/root/webapps/geoserver/WEB-INF/lib/gt-jdbc-postgis-26.2.jar org.geotools.gce.imagemosaic.jdbc.Toolbox import -config dgm.postgis.xml -spatialTNPrefix dgm -tileTNPrefix dgm -dir tiles/ -ext tif
mkdir /root/data_dir/coverage/dgm
cp *.xml* /root/data_dir/coverages/dgm

Finally, I configured the datastore with the browser client. For small raster tables this works well. But for bigger tables I receive the following error:

Failed to create reader from file:coverages/dgm/dgm.postgis.xml and hints Hints: STYLE_FACTORY = StyleFactoryImpl FORCE_LONGITUDE_FIRST_AXIS_ORDER = true LENIENT_DATUM_SHIFT = true GRID_COVERAGE_FACTORY = GridCoverageFactory TILE_ENCODING = null REPOSITORY = org.geoserver.catalog.CatalogRepository@66c9b911 FILTER_FACTORY = FilterFactoryImpl FORCE_AXIS_ORDER_HONORING = http COMPARISON_TOLERANCE = 1.0E-8 EXECUTOR_SERVICE = java.util.concurrent.ThreadPoolExecutor@4137b934Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0 FEATURE_FACTORY = org.geotools.feature.LenientFeatureFactoryImpl@44de94c3

I already increased the jvm heap size. Nevertheless, I am getting the error above. Can someone help me, how I need to configure the geoserver, that I can work with it, also with big raster tiles, stored in a postgresql database?

For every hint I am grateful.

Best Regards

EDIT 1: After setting in global setting the log-level to developer I receive the following output message in the browser:

Failed to create reader from file:coverages/dgm/dgm.postgis.xml and hints Hints: STYLE_FACTORY = StyleFactoryImpl FORCE_LONGITUDE_FIRST_AXIS_ORDER = true LENIENT_DATUM_SHIFT = true GRID_COVERAGE_FACTORY = GridCoverageFactory TILE_ENCODING = null REPOSITORY = org.geoserver.catalog.CatalogRepository@360a7fe7 FILTER_FACTORY = FilterFactoryImpl FORCE_AXIS_ORDER_HONORING = http COMPARISON_TOLERANCE = 1.0E-8 EXECUTOR_SERVICE = java.util.concurrent.ThreadPoolExecutor@65d3cd0bRunning, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0 FEATURE_FACTORY = org.geotools.feature.LenientFeatureFactoryImpl@44de94c3

The log file looks like this:

2022-03-03 20:33:32,429 TRACE org.geoserver.platform.resource - polling contents of /root/data_dir/gwc-layers 2022-03-03 20:33:36,632 TRACE org.geoserver.platform.resource - delta computed in 3,878ms for /root/data_dir/gwc-layers 2022-03-03 20:33:41,459 ERROR org.geotools.gce.imagemosaic.jdbc - Ran out of memory retrieving query results. org.postgresql.util.PSQLException: Ran out of memory retrieving query results. at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2256) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:323) at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:481) at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:401) at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:164) at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:114) at org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96) at org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96) at org.geotools.gce.imagemosaic.jdbc.JDBCAccessBase.getPixelResolution(JDBCAccessBase.java:696) at org.geotools.gce.imagemosaic.jdbc.JDBCAccessBase.calculateResolutionsFromDB(JDBCAccessBase.java:433) at org.geotools.gce.imagemosaic.jdbc.JDBCAccessBase.initialize(JDBCAccessBase.java:121) at org.geotools.gce.imagemosaic.jdbc.JDBCAccessFactory.getJDBCAcess(JDBCAccessFactory.java:90) at org.geotools.gce.imagemosaic.jdbc.ImageMosaicJDBCReader.<init>(ImageMosaicJDBCReader.java:144) at org.geotools.gce.imagemosaic.jdbc.ImageMosaicJDBCFormat.getReader(ImageMosaicJDBCFormat.java:197) at org.geotools.gce.imagemosaic.jdbc.ImageMosaicJDBCFormat.getReader(ImageMosaicJDBCFormat.java:59) at org.geoserver.catalog.ResourcePool.getGridCoverageReader(ResourcePool.java:1573) at org.geoserver.catalog.ResourcePool.getGridCoverageReader(ResourcePool.java:1518) at org.geoserver.catalog.ResourcePool.getGridCoverageReader(ResourcePool.java:1505)

. . . cut some log messages

at

org.apache.wicket.ajax.form.AjaxFormSubmitBehavior$AjaxFormSubmitter.onSubmit(AjaxFormSubmitBehavior.java:218) at org.apache.wicket.markup.html.form.Form.delegateSubmit(Form.java:1312) at org.apache.wicket.markup.html.form.Form.process(Form.java:976) at org.apache.wicket.markup.html.form.Form.onFormSubmitted(Form.java:797) at org.apache.wicket.ajax.form.AjaxFormSubmitBehavior.onEvent(AjaxFormSubmitBehavior.java:174) at org.apache.wicket.ajax.AjaxEventBehavior.respond(AjaxEventBehavior.java:155) at org.apache.wicket.ajax.AbstractDefaultAjaxBehavior.onRequest(AbstractDefaultAjaxBehavior.java:601) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 2022-03-03 20:33:41,460 ERROR org.geotools.gce.imagemosaic.jdbc - Ran out of memory retrieving query results. 2022-03-03 20:33:41,461 ERROR org.geotools.gce.imagemosaic.jdbc - org.postgresql.util.PSQLException: Ran out of memory retrieving query results. 2022-03-03 20:33:41,461 WARN org.geotools.gce.imagemosaic.jdbc - java.io.IOException: org.postgresql.util.PSQLException: Ran out of memory retrieving query results. java.io.IOException: java.io.IOException: org.postgresql.util.PSQLException: Ran out of memory retrieving query results. at org.geotools.gce.imagemosaic.jdbc.ImageMosaicJDBCReader.<init>(ImageMosaicJDBCReader.java:147) at org.geotools.gce.imagemosaic.jdbc.ImageMosaicJDBCFormat.getReader(ImageMosaicJDBCFormat.java:197) at org.geotools.gce.imagemosaic.jdbc.ImageMosaicJDBCFormat.getReader(ImageMosaicJDBCFormat.java:59) at org.geoserver.catalog.ResourcePool.getGridCoverageReader(ResourcePool.java:1573) at org.geoserver.catalog.ResourcePool.getGridCoverageReader(ResourcePool.java:1518) at org.geoserver.catalog.ResourcePool.getGridCoverageReader(ResourcePool.java:1505) at org.geoserver.web.data.store.CoverageStoreEditPage.onSave(CoverageStoreEditPage.java:126) at org.geoserver.web.data.store.AbstractCoverageStorePage$1.onSubmit(AbstractCoverageStorePage.java:138) at org.apache.wicket.ajax.markup.html.form.AjaxSubmitLink$1.onSubmit(AjaxSubmitLink.java:111) at org.apache.wicket.ajax.form.AjaxFormSubmitBehavior$AjaxFormSubmitter.onSubmit(AjaxFormSubmitBehavior.java:218) at org.apache.wicket.markup.html.form.Form.delegateSubmit(Form.java:1312) at org.apache.wicket.markup.html.form.Form.process(Form.java:976)

. . . cut some log messages

at

org.geoserver.catalog.ResourcePool.getGridCoverageReader(ResourcePool.java:1518) at org.geoserver.catalog.ResourcePool.getGridCoverageReader(ResourcePool.java:1505) at org.geoserver.web.data.store.CoverageStoreEditPage.onSave(CoverageStoreEditPage.java:126) at org.geoserver.web.data.store.AbstractCoverageStorePage$1.onSubmit(AbstractCoverageStorePage.java:138) at org.apache.wicket.ajax.markup.html.form.AjaxSubmitLink$1.onSubmit(AjaxSubmitLink.java:111) at org.apache.wicket.ajax.form.AjaxFormSubmitBehavior$AjaxFormSubmitter.onSubmit(AjaxFormSubmitBehavior.java:218) at org.apache.wicket.markup.html.form.Form.delegateSubmit(Form.java:1312) at org.apache.wicket.markup.html.form.Form.process(Form.java:976) at org.apache.wicket.markup.html.form.Form.onFormSubmitted(Form.java:797) at org.apache.wicket.ajax.form.AjaxFormSubmitBehavior.onEvent(AjaxFormSubmitBehavior.java:174) at org.apache.wicket.ajax.AjaxEventBehavior.respond(AjaxEventBehavior.java:155) at org.apache.wicket.ajax.AbstractDefaultAjaxBehavior.onRequest(AbstractDefaultAjaxBehavior.java:601) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 2022-03-03 20:33:41,517 DEBUG org.geoserver - Thread 16 locking in mode READ 2022-03-03 20:33:41,517 DEBUG org.geoserver - Thread 16 releasing the lock in mode READ 2022-03-03 20:33:41,583 DEBUG org.geoserver.filters - Compressing output for mimetype: text/xml;charset=utf-8 2022-03-03 20:33:41,584 DEBUG org.geoserver.security.filter.GeoServerSecurityContextPersistenceFilter$1

EDIT 2: adjustment of the jvm heap size

JAVA_OPTS="-Xms20000m -Xmx20000m"

-- skynet1010
docker
geoserver
jvm
kubernetes
postgresql

Similar Questions

kube-dns getsockopt no route to host
go gRPC routing without specifying port
Kubernetes liveness - Reserve threads/memory for a specific endpoint with Spring Boot
kubernetes nginx-ingress controller Liveness probe failed
Need advice how to make Spinnaker work with aws ecr?

0 Answers