Error when using matrix in batch mode

[sigsdis30]

Hi,

I use Matrix tool (ORS Tools v1.5.2 with QGIS 3.16.4 or 3.22.4) in batch mode.

When I enter parameters manually, only the first process fails. But, the same process works fine if I execute in a stand alone operation. When I upload a json file, all processes fail. Parameters are identical, except INPUT_START_LAYER and INPUT_END_LAYER (layer's name is automatically replaced by the database connection string in json file).
Last week, I successfully do the same task with the same tables and the same parameters. So, I don't understand...

Output files are created but some rows are missing.

Thank you for your help

I get this message when entering the parameters manually:

---

Algorithme de traitement 1/31…
Démarrage de l'algorithme Matrix from layers…
Paramètres en entrée:
{'INPUT_AVOID_BORDERS': None,
'INPUT_AVOID_COUNTRIES': '',
'INPUT_AVOID_FEATURES': [],
'INPUT_AVOID_POLYGONS': None,
'INPUT_END_FIELD': 'esz',
'INPUT_END_LAYER': 'zec_id_1_a_3499_v_eb4b4925_ef3b_496a_80c7_8246bbe43602',
'INPUT_PROFILE': 0,
'INPUT_PROVIDER': 0,
'INPUT_START_FIELD': 'code',
'INPUT_START_LAYER': 'cis_als_v_33dfe97a_e7d4_424f_bd08_49bc1709fdca',
'OUTPUT': <QgsProcessingOutputLayerDefinition {'sink':L:/PREVISION/SIG/bd_opensis/ro/matrices/vl_h_1_cis_als_v.csv, 'createOptions': {}}>}

Traceback (most recent call last):
File "C:/Users/v-n/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\ORStools\proc\matrix_proc.py", line 189, in processAlgorithm
duration = response['durations'][s][d]
IndexError: list index out of range

L'algorithme Matrix from layers a échoué…
Execution failed after 2.75 seconds

---

I get this message when I upload json file (I just deleted DB connection parameters):

---

Algorithme de traitement 1/31…
Démarrage de l'algorithme Matrix from layers…
Paramètres en entrée:
{'INPUT_AVOID_BORDERS': None,
'INPUT_AVOID_COUNTRIES': '',
'INPUT_AVOID_FEATURES': [],
'INPUT_AVOID_POLYGONS': None,
'INPUT_END_FIELD': 'esz',
'INPUT_END_LAYER': "postgres://dbname='lizmap_opensis_sis30' "
'host=***** port=***** '
"user='' password='' "
"sslmode=disable key='id' estimatedmetadata=true srid=2154 "
"type=Point checkPrimaryKeyUnicity='1' "
'table="ro"."zec_id_1_a_3499_v" (geom_2154)',
'INPUT_PROFILE': 0,
'INPUT_PROVIDER': 0,
'INPUT_START_FIELD': 'code',
'INPUT_START_LAYER': "postgres://dbname='lizmap_opensis_sis30' "
'host=***** port=***** '
"user='' password='' "
"sslmode=disable key='code' estimatedmetadata=true "
"srid=2154 type=Point checkPrimaryKeyUnicity='1' "
'table="sdis"."cis_als_v" (geom_2154)',
'OUTPUT': <QgsProcessingOutputLayerDefinition {'sink':L:/PREVISION/SIG/bd_opensis/ro/matrices/vl_h_1_cis_als_v.csv, 'createOptions': {}}>}

Traceback (most recent call last):
File "C:/Users/v-n/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\ORStools\proc\matrix_proc.py", line 189, in processAlgorithm
duration = response['durations'][s][d]
IndexError: list index out of range

L'algorithme Matrix from layers a échoué…
Execution failed after 3.84 seconds

[merydian]

Hello @sigsdis30,
thank you for posting this issue and contributing to the plugin. Could you provide us with more information? It is difficult to tell what the problem is based on what we have here.

Best regards

[sigsdis30]

Hello,

What kind of information do you require? I'll do my best to give them to you

[merydian]

If you could provide us with the exported layers and the batch-json, it would probably be of help. Is it possible you upload them here, without sensitive information of course?

[sigsdis30]

Thank you very much. I've just got back to this subject. I'm finding it difficult to reproduce this behaviour systematically. The following command works but not always. Sometimes it returns an error but the output CSV is generated, correctly I think. It corresponds to what is executed in batch mode. In this mode, the execution can be correct for several entries and get stuck on a particular entry, not always the same one. By entering the rows manually or loading a JSON template, the behaviour is identical.
Thanks for your help

{"INPUT_END_FIELD":"id","INPUT_END_LAYER":"dbname='cassini' host=*************** port=**** sslmode=disable authcfg=******* key='id' estimatedmetadata=true srid=2154 type=Point checkPrimaryKeyUnicity='1' table="dni"."rop_zec_surfacique" (point_2154) sql="etat_de_l_objet" = 'En service' AND "id" < 2500","INPUT_PROFILE":0,"INPUT_PROVIDER":0,"INPUT_START_FIELD":"id","INPUT_START_LAYER":"dbname='cassini' host=*************** port=**** sslmode=disable authcfg=******* key='id' estimatedmetadata=true srid=2154 type=Point checkPrimaryKeyUnicity='1' table="dni"."org_unite_operationnelle" (geom_2154) sql="code" = 'AIM' AND "etat_de_l_objet" = 'Projeté' AND "code_du_departement" = '30'","OUTPUT":"U:/traitements/gestion_rop/matrices/aimp_to_zec2500.csv"}

[sigsdis30]

Here is an extract from the JSON file, whose syntax varies slightly:
[{"PARAMETERS": { "INPUT_PROVIDER": "0", "INPUT_PROFILE": "0", "INPUT_START_LAYER": "'postgres://dbname=\\'cassini\\' host=*************** port=**** sslmode=disable authcfg=******* key=\\'id\\' estimatedmetadata=true srid=2154 type=Point checkPrimaryKeyUnicity=\\'1\\' table=\"dni\".\"org_unite_operationnelle\" (geom_2154) sql=\"code\" = \\'AIM\\' AND \"etat_de_l_objet\" = \\'Projet\u00e9\\' AND \"code_du_departement\" = \\'30\\''", "INPUT_START_FIELD": "'id'", "INPUT_END_LAYER": "'postgres://dbname=\\'cassini\\' host=*************** port=**** sslmode=disable authcfg=******* key=\\'id\\' estimatedmetadata=true srid=2154 type=Point checkPrimaryKeyUnicity=\\'1\\' table=\"dni\".\"rop_zec_surfacique\" (point_2154) sql=\"etat_de_l_objet\" = \\'En service\\' AND \"id\" < 2500'", "INPUT_END_FIELD": "'id'", "INPUT_AVOID_FEATURES": "[]", "INPUT_AVOID_BORDERS": "None", "INPUT_AVOID_COUNTRIES": "''", "INPUT_AVOID_POLYGONS": "None"}, "OUTPUTS": {"OUTPUT": "U:/traitements/gestion_rop/matrices/aimp_to_zec2500.csv"}}, {"PARAMETERS": { "INPUT_PROVIDER": "0", "INPUT_PROFILE": "0", "INPUT_START_LAYER": "'postgres://dbname=\\'cassini\\' host=*************** port=**** sslmode=disable authcfg=******* key=\\'id\\' estimatedmetadata=true srid=2154 type=Point checkPrimaryKeyUnicity=\\'1\\' table=\"dni\".\"org_unite_operationnelle\" (geom_2154) sql=\"code\" = \\'ALS\\' AND \"etat_de_l_objet\" = \\'Actif\\' AND \"code_du_departement\" = \\'30\\''", "INPUT_START_FIELD": "'id'", "INPUT_END_LAYER": "'postgres://dbname=\\'cassini\\' host=*************** port=**** sslmode=disable authcfg=******* key=\\'id\\' estimatedmetadata=true srid=2154 type=Point checkPrimaryKeyUnicity=\\'1\\' table=\"dni\".\"rop_zec_surfacique\" (point_2154) sql=\"etat_de_l_objet\" = \\'En service\\' AND \"id\" < 2500'", "INPUT_END_FIELD": "'id'", "INPUT_AVOID_FEATURES": "[]", "INPUT_AVOID_BORDERS": "None", "INPUT_AVOID_COUNTRIES": "''", "INPUT_AVOID_POLYGONS": "None"}, "OUTPUTS": {"OUTPUT": "U:/traitements/gestion_rop/matrices/als_to_zec2500.csv"}} ]

[sigsdis30]

I've just run the command after loading my JSON file. Here's the error message:
Traceback (most recent call last):
File "C:\Users/********/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\ORStools\proc\matrix_proc.py", line 194, in processAlgorithm
duration = response["durations"][s][d]
IndexError: list index out of range

CSV files are generated but the last line inside is missing (normally the INPUT_END_LAYER layer should contain all id < 2500 but
id=2499 is missing). I've attached an example.
I hope I've helped.

aimp_to_zec2500.csv
als_to_zec2500.csv

[sigsdis30]

One last thing. I have just run the calculation with the following parameters:
{ 'INPUT_END_FIELD' : 'id', 'INPUT_END_LAYER' : 'postgres://dbname='cassini' host=*************** port=**** sslmode=disable authcfg=******* key='id' estimatedmetadata=true srid=2154 type=Point checkPrimaryKeyUnicity='1' table="dni"."rop_zec_surfacique" (point_2154) sql="etat_de_l_objet" = 'En service' AND "id" < 2500', 'INPUT_PROFILE' : 0, 'INPUT_PROVIDER' : 0, 'INPUT_START_FIELD' : 'id', 'INPUT_START_LAYER' : 'postgres://dbname='cassini' host=*************** port=**** sslmode=disable authcfg=******* key='id' estimatedmetadata=true srid=2154 type=Point checkPrimaryKeyUnicity='1' table="dni"."org_unite_operationnelle" (geom_2154) sql="code" = 'AIM' AND "etat_de_l_objet" = 'Projeté' AND "code_du_departement" = '30'', 'OUTPUT' : 'TEMPORARY_OUTPUT' }

The result is correct, in batch mode (creating the rows by hand) or not: I have the id = 2499 in the result

[merydian]

Hi sigsdis30

I've had a look into this. The problem can be reproduced by popping one of the response items. That means the length of the API response is shorter that expected. Is there anything that may cause a route to not be created?

Are you still using ORS Tools v1.5.2, or the latest version? Are you using the public provider?

[sigsdis30]

Hi Merydian,
Initially, I was using version 1.5.2, then I updated it. As I had errors with the new version, I reinstalled version 1.5.2. Recently, I needed to regenerate some matrices and, seeing that version 1.7.1 was available, I wanted to try using it. This is the one I tested with.
In all cases, I used the public provider.
I don't know what could prevent a route from being created. What I do know is that some destinations cannot be reached. For the records concerned, the "duration" and "dist_km" values are simply zero. In batch mode, the problem doesn't always occur: the calculation may fail the first time and succeed the second. As I try to generate a file for each starting point, it may succeed for some points and fail for others. For the starting point that failed, the file is generated but the last line is still missing, i.e. the id of the last arrival point. For example, if I generate the matrix from a given start point to all the end points from 1 to 2499, the id 2499 will always be missing in the event of a crash. Could it be that the process can't write the end of the file?

[koebi]

Hey,

this is about impossible to debug without having an idea of what your input looks like.
Could you provide an input file that exhibits the issue?

Best regards

[sigsdis30]

Hello,
The input data are two PostgreSQL tables that you cannot access. Their structure is as follows:

• id serial
• text code (used in INPUT_START_FIELD and INPUT_END_FIELD)
• geom(Point,2154) (simple point geometry and EPSG:2154)
  It's hard to get much more basic than this.

[koebi]

Hey,

we assume that your issue is related to the locations of the points that you're using to request your matrix with.
Thus, reproducing your issue is only possible if we have access to the geometries of the points that exhibit the problem on your side.

Could you export ids and geometry from your PostgreSQL tables and provide that?

If not, can you provide artificial data that exhibits the same problem?

Thanks for your help and patience :)

[merydian]

Hi sigsdis30,

may it be the case that u are using too many points at once? The restriction for the matrix API is 3500 Locations per request, see here. You can check the log after the error appears, at View>Panels>Log Messages>ORS Tools to see if this is happening.

[sigsdis30]

Hello,
Thank you for taking the time to help me.
I'm attaching a geopackage containing an extract of the starting points layer and the destination points layer: https://we.tl/t-rT2tHXZTvb
There's no problem with over-quota queries, because I'm creating table extracts to make sure I don't exceed them: one view per starting point and one view per batch of 2,500 destination points. I'm running the whole thing in batch mode, which is particularly useful for managing the timeout when the number of authorised queries has been exceeded (a one-minute wait, I think, before the next burst).

[koebi]

Hey,

thanks for the data.

When I use this to request a matrix, I do get the error mentioned by @merydian - the restriction is 3500 routes per request, and 2x2500 exceeds this.

I would guess that this is the underlying issue in your case as well - could you try running without views, but rather with extracted points?
The case of running with views on your psql db is quite hard to debug, since we don't have your db and views…

Please also have a look at the ORS Tools Log Messages Panel, as proposed by @merydian , to see what issue the ors has with your query.

Best regards

[sigsdis30]

You need to run separate queries for each starting point. If you don't do this, you'll exceed the quota.
Views make data management easier, but even without them, the problem occurs. I don't agree with your hypothesis

[sigsdis30]

Some news?
Is it possible that the JSON syntax returned by the batch run is incorrect?
Best regards