chore: Update hierarchy query library #1223

bprusinowski · 2023-10-17T13:43:57Z

This PR updates the cube-hierarchy-query library to v2.2.0.

It also updates the way of how we fetch dimension values metadata for hierarchies by using new syntax to directly specify the properties we need to fetch along the hierarchy. However, this seem to make it significantly less performant that our current approach, which means we should probably only update library, but keep the old way of fetching things.

New auto-generated query used to fetch hierarchies and metadata (NFI dataset) – approx. 15s

PREFIX schema: <http://schema.org/>
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>

CONSTRUCT { 
  ?resource1 schema:identifier ?resource2 .
  ?resource1 schema:name ?resource3 .
  ?resource1 schema:description ?resource5 .
  ?resource1 schema:position ?resource7 .
  ?resource1 schema:alternateName ?resource8 .
  ?resource1 rdf:type ?resource10 .
  ?resource11 schema:containedInPlace ?resource1 .
  ?resource11 schema:identifier ?resource12 .
  ?resource11 schema:name ?resource13 .
  ?resource11 schema:description ?resource15 .
  ?resource11 schema:position ?resource17 .
  ?resource11 schema:alternateName ?resource18 .
  ?resource11 rdf:type ?resource20 .
}
WHERE {
  VALUES ( ?resource1 ) {
    ( <https://ld.admin.ch/country/CHE> )
  }
  {   
    ?resource1 schema:identifier ?resource2 .
  } UNION {
  {
    VALUES ( ?resource1 ) {
     ( <https://ld.admin.ch/country/CHE> )
    }
    ?resource1 schema:name ?resource3 .
    FILTER (lang(?resource3) = "de" ) }
  } UNION {    
{ 
VALUES ( ?resource1 )
{
( <https://ld.admin.ch/country/CHE> )
}

      ?resource1 schema:description ?resource5 .
      FILTER (lang(?resource5) = "de" ) }

    } UNION {
      
?resource1 schema:position ?resource7 .

    } UNION {
      
{ 
VALUES ( ?resource1 )
{
( <https://ld.admin.ch/country/CHE> )
}

      ?resource1 schema:alternateName ?resource8 .
      FILTER (lang(?resource8) = "de" ) }

    } UNION {
      
?resource1 rdf:type ?resource10 .

    } UNION {
      { 
    
?resource11 schema:containedInPlace ?resource1 .
 
  } UNION {
      
            
?resource11 schema:containedInPlace ?resource1 .

            

{ 
    
?resource11 schema:identifier ?resource12 .
 
  } UNION {
      
{ 
VALUES ( ?resource1 )
{
( <https://ld.admin.ch/country/CHE> )
}

?resource11 schema:containedInPlace ?resource1 .



      ?resource11 schema:name ?resource13 .
      FILTER (lang(?resource13) = "de" ) }

    } UNION {
      
{ 
VALUES ( ?resource1 )
{
( <https://ld.admin.ch/country/CHE> )
}

?resource11 schema:containedInPlace ?resource1 .



      ?resource11 schema:description ?resource15 .
      FILTER (lang(?resource15) = "de" ) }

    } UNION {
      
?resource11 schema:position ?resource17 .

    } UNION {
      
{ 
VALUES ( ?resource1 )
{
( <https://ld.admin.ch/country/CHE> )
}

?resource11 schema:containedInPlace ?resource1 .



      ?resource11 schema:alternateName ?resource18 .
      FILTER (lang(?resource18) = "de" ) }

    } UNION {
      
?resource11 rdf:type ?resource20 .

    }

          
    }
    }


      ?resource1 ^schema:containedInPlace ?resource11 .
      ?resource11 a <https://environment.ld.admin.ch/foen/nfi/UnitOfReference/827> .
    
  
}

Old auto-generated query to fetch hierarchies ((NFI dataset) ) – approx. 2s

PREFIX schema: <http://schema.org/>

CONSTRUCT { 
    ?resource2 schema:containedInPlace ?resource1 .
?resource2 ?resource3 ?resource4 .
?resource1 ?resource5 ?resource6 .
   }

WHERE {

    

VALUES ( ?resource1 )
{
( <https://ld.admin.ch/country/CHE> )
}
{ 
    { 
    
?resource2 schema:containedInPlace ?resource1 .
 
  } UNION {
      
            
?resource2 schema:containedInPlace ?resource1 .

            


        ?resource2 ?resource3 ?resource4 .
        
        
      

          
    } 
  } UNION {
      
        ?resource1 ?resource5 ?resource6 .
        
        
      
    }


      ?resource1 ^schema:containedInPlace ?resource2 .
      ?resource2 a <https://environment.ld.admin.ch/foen/nfi/UnitOfReference/827> .
    
  
}

vercel · 2023-10-17T13:44:03Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
visualization-tool	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Nov 15, 2023 8:29am

Rdataflow · 2023-10-17T16:31:00Z

@bprusinowski unfortunately https://visualization-tool-am4klf8am-ixt1.vercel.app/create/new?cube=https://environment.ld.admin.ch/foen/nfi/nfi_C-90/cube/2023-2&dataSource=Prod fails to load with this changes... there might be something important missing yet

bprusinowski · 2023-10-18T14:00:16Z

@Rdataflow, this is related to the fact that serverless functions for apps deployed on Vercel can execute for max. 15s, while the request was supposed to take longer.

This is related to a fact that by providing a list of properties to fetch along with hierarchy made it less performant, I've updated the description of the PR.

Rdataflow · 2023-10-18T16:37:09Z

@bprusinowski hmm this would be the contrary of the performance improvements to strive for...
maybe adrian has input on this generated query?
wanna retry and omit the lang filters?

nb: @bprusinowski don't merge this PR yet... I'll tell you more

Rdataflow · 2023-10-18T17:04:48Z

for the records:
concerned GQL query

curl https://test.visualize.admin.ch/api/graphql -X POST -H 'Content-Type: application/json' -H 'x-visualize-cache-control: no-cache' --data-raw '{"operationName":"ComponentsWithHierarchies","variables":{"iri":"https://environment.ld.admin.ch/foen/nfi/nfi_C-90/cube/2023-2","sourceType":"sparql","sourceUrl":"https://lindas.admin.ch/query","locale":"de"},"query":"query ComponentsWithHierarchies($iri: String!, $sourceType: String!, $sourceUrl: String!, $locale: String!, $latest: Boolean, $filters: Filters, $componentIris: [String!]) {\n  dataCubeByIri(\n    iri: $iri\n    sourceType: $sourceType\n    sourceUrl: $sourceUrl\n    locale: $locale\n    latest: $latest\n  ) {\n    dimensions(\n      sourceType: $sourceType\n      sourceUrl: $sourceUrl\n      componentIris: $componentIris\n    ) {\n      ...dimensionMetadataWithHierarchies\n      __typename\n    }\n    measures(\n      sourceType: $sourceType\n      sourceUrl: $sourceUrl\n      componentIris: $componentIris\n    ) {\n      ...dimensionMetadataWithHierarchies\n      __typename\n    }\n    __typename\n  }\n}\n\nfragment dimensionMetadataWithHierarchies on Dimension {\n  iri\n  label\n  description\n  isNumerical\n  isKeyDimension\n  scaleType\n  dataType\n  order\n  values(sourceType: $sourceType, sourceUrl: $sourceUrl, filters: $filters)\n  unit\n  related {\n    iri\n    type\n    __typename\n  }\n  ... on TemporalDimension {\n    timeUnit\n    timeFormat\n    __typename\n  }\n  ... on NumericalMeasure {\n    isCurrency\n    currencyExponent\n    resolution\n    isDecimal\n    __typename\n  }\n  ...hierarchyMetadata\n}\n\nfragment hierarchyMetadata on Dimension {\n  hierarchy(sourceType: $sourceType, sourceUrl: $sourceUrl) {\n    ...hierarchyValueFields\n    children {\n      ...hierarchyValueFields\n      children {\n        ...hierarchyValueFields\n        children {\n          ...hierarchyValueFields\n          children {\n            ...hierarchyValueFields\n            children {\n              ...hierarchyValueFields\n              __typename\n            }\n            __typename\n          }\n          __typename\n        }\n        __typename\n      }\n      __typename\n    }\n    __typename\n  }\n}\n\nfragment hierarchyValueFields on HierarchyValue {\n  value\n  dimensionIri\n  depth\n  label\n  alternateName\n  hasValue\n  position\n  identifier\n}\n"}' -s -o /dev/null -w '%{time_total}'

bprusinowski · 2023-10-19T12:57:35Z

@Rdataflow query without language filters is much faster than 15s (takes around 0.5s) :)

PREFIX schema: <http://schema.org/>
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>

CONSTRUCT { 
    ?resource1 schema:identifier ?resource2 .
?resource1 schema:name ?resource3 .
?resource1 schema:description ?resource4 .
?resource1 schema:position ?resource5 .
?resource1 schema:alternateName ?resource6 .
?resource1 rdf:type ?resource7 .
?resource8 schema:containedInPlace ?resource1 .
?resource8 schema:identifier ?resource9 .
?resource8 schema:name ?resource10 .
?resource8 schema:description ?resource11 .
?resource8 schema:position ?resource12 .
?resource8 schema:alternateName ?resource13 .
?resource8 rdf:type ?resource14 .
   }

WHERE {

    

VALUES ( ?resource1 )
{
( <https://ld.admin.ch/country/CHE> )
}
{ 
    
?resource1 schema:identifier ?resource2 .
 
  } UNION {
      
?resource1 schema:name ?resource3 .

    } UNION {
      
?resource1 schema:description ?resource4 .

    } UNION {
      
?resource1 schema:position ?resource5 .

    } UNION {
      
?resource1 schema:alternateName ?resource6 .

    } UNION {
      
?resource1 rdf:type ?resource7 .

    } UNION {
      { 
    
?resource8 schema:containedInPlace ?resource1 .
 
  } UNION {
      
            
?resource8 schema:containedInPlace ?resource1 .

            

{ 
    
?resource8 schema:identifier ?resource9 .
 
  } UNION {
      
?resource8 schema:name ?resource10 .

    } UNION {
      
?resource8 schema:description ?resource11 .

    } UNION {
      
?resource8 schema:position ?resource12 .

    } UNION {
      
?resource8 schema:alternateName ?resource13 .

    } UNION {
      
?resource8 rdf:type ?resource14 .

    }

          
    }
    }


      ?resource1 ^schema:containedInPlace ?resource8 .
      ?resource8 a <https://environment.ld.admin.ch/foen/nfi/UnitOfReference/49> .
    
  
}

Rdataflow · 2023-10-19T13:06:43Z

@bprusinowski good to know, can you share your slow experience and the probable reason here to keep awareness:
https://gitlab.ldbar.ch/bafu/visualize/-/issues/548
THX

Rdataflow · 2023-11-02T18:26:13Z

@bprusinowski can you rebase this PR to main?

bprusinowski · 2023-11-02T18:45:11Z

Hey @Rdataflow, the base is actually main. Did you mean to merge it to main?

Rdataflow · 2023-11-03T10:56:54Z

Hey @Rdataflow, the base is actually main. Did you mean to merge it to main?

I just would like to see the latest commits from main branch included for the preview, thus only rebase on main trunk...

…into chore/update-hierarchy-query-library

bprusinowski · 2023-11-03T13:46:10Z

Ah I see, done 👍

Rdataflow · 2023-11-03T18:05:44Z

@bprusinowski v2.1.1 is out now 🚀 let's give it a try

bprusinowski · 2023-11-08T11:09:45Z

@Rdataflow it looks that the performance is still not ideal when using language filters; when removed, it gets much better. Let me know if we should simply switch to querying all languages and extracting them on our own 👀

Generated query (not visible in deployment preview due to >15s execution time)

PREFIX schema: <http://schema.org/>
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>

CONSTRUCT { 
    ?resource1 schema:identifier ?resource2 .
?resource1 schema:name ?resource3 .
?resource1 schema:description ?resource5 .
?resource1 schema:position ?resource7 .
?resource1 schema:alternateName ?resource8 .
?resource1 rdf:type ?resource10 .
?resource11 schema:containedInPlace ?resource1 .
?resource11 schema:identifier ?resource12 .
?resource11 schema:name ?resource13 .
?resource11 schema:description ?resource15 .
?resource11 schema:position ?resource17 .
?resource11 schema:alternateName ?resource18 .
?resource11 rdf:type ?resource20 .
   }

WHERE {

    

VALUES ( ?resource1 )
{
( <https://ld.admin.ch/country/CHE> )
}
{ 
    
?resource1 schema:identifier ?resource2 .
 
  } UNION {
      
{ 
VALUES ( ?resource1 )
{
( <https://ld.admin.ch/country/CHE> )
}

      ?resource1 schema:name ?resource3 .
      FILTER (lang(?resource3) = "en" ) }

    } UNION {
      
{ 
VALUES ( ?resource1 )
{
( <https://ld.admin.ch/country/CHE> )
}

      ?resource1 schema:description ?resource5 .
      FILTER (lang(?resource5) = "en" ) }

    } UNION {
      
?resource1 schema:position ?resource7 .

    } UNION {
      
{ 
VALUES ( ?resource1 )
{
( <https://ld.admin.ch/country/CHE> )
}

      ?resource1 schema:alternateName ?resource8 .
      FILTER (lang(?resource8) = "en" ) }

    } UNION {
      
?resource1 rdf:type ?resource10 .

    } UNION {
      { 
    
?resource11 schema:containedInPlace ?resource1 .
 
  } UNION {
      
            
?resource11 schema:containedInPlace ?resource1 .

            

{ 
    
?resource11 schema:identifier ?resource12 .
 
  } UNION {
      
{ 
VALUES ( ?resource1 )
{
( <https://ld.admin.ch/country/CHE> )
}

?resource11 schema:containedInPlace ?resource1 .



      ?resource11 schema:name ?resource13 .
      FILTER (lang(?resource13) = "en" ) }

    } UNION {
      
{ 
VALUES ( ?resource1 )
{
( <https://ld.admin.ch/country/CHE> )
}

?resource11 schema:containedInPlace ?resource1 .



      ?resource11 schema:description ?resource15 .
      FILTER (lang(?resource15) = "en" ) }

    } UNION {
      
?resource11 schema:position ?resource17 .

    } UNION {
      
{ 
VALUES ( ?resource1 )
{
( <https://ld.admin.ch/country/CHE> )
}

?resource11 schema:containedInPlace ?resource1 .



      ?resource11 schema:alternateName ?resource18 .
      FILTER (lang(?resource18) = "en" ) }

    } UNION {
      
?resource11 rdf:type ?resource20 .

    }

          
    }
    }


      ?resource1 ^schema:containedInPlace ?resource11 .
      ?resource11 a <https://environment.ld.admin.ch/foen/nfi/UnitOfReference/49> .
    
  
}

bprusinowski · 2024-02-23T15:45:23Z

Closed in favor of #1364.

bprusinowski requested a review from ptbrowne as a code owner October 17, 2023 13:43

vercel bot deployed to Preview October 17, 2023 13:48 View deployment

bprusinowski added 3 commits October 18, 2023 16:06

perf: Async fetch metadata and hierarchies

4c7c5f3

chore: Update hierarchy library

f711dd0

refactor: Only fetch necessary properties when fetching hierarchy

a102008

bprusinowski force-pushed the chore/update-hierarchy-query-library branch from 9fcc4aa to a102008 Compare October 18, 2023 14:06

vercel bot deployed to Preview October 18, 2023 14:10 View deployment

Merge branch 'main' of github.com:visualize-admin/visualization-tool …

7cae9da

…into chore/update-hierarchy-query-library

vercel bot deployed to Preview November 3, 2023 13:49 View deployment

chore: Update cube-hierarchy-query to v2.1.1

0191433

vercel bot deployed to Preview November 8, 2023 10:56 View deployment

bprusinowski mentioned this pull request Nov 14, 2023

feat: Add a way to retrieve dimensions from multiple cubes #1244

Merged

3 tasks

bprusinowski added 2 commits November 15, 2023 09:22

chore: Update cube-hierarchy-query to v2.2.0

87cfaa7

perf: Add pragmas to hierarchy query

f4079d0

vercel bot deployed to Preview November 15, 2023 08:29 View deployment

bprusinowski closed this Feb 23, 2024

bprusinowski deleted the chore/update-hierarchy-query-library branch March 25, 2024 09:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: Update hierarchy query library #1223

chore: Update hierarchy query library #1223

bprusinowski commented Oct 17, 2023 •

edited

Loading

vercel bot commented Oct 17, 2023 •

edited

Loading

Rdataflow commented Oct 17, 2023

bprusinowski commented Oct 18, 2023

Rdataflow commented Oct 18, 2023

Rdataflow commented Oct 18, 2023

bprusinowski commented Oct 19, 2023 •

edited

Loading

Rdataflow commented Oct 19, 2023

Rdataflow commented Nov 2, 2023

bprusinowski commented Nov 2, 2023

Rdataflow commented Nov 3, 2023

bprusinowski commented Nov 3, 2023

Rdataflow commented Nov 3, 2023

bprusinowski commented Nov 8, 2023

bprusinowski commented Feb 23, 2024

chore: Update hierarchy query library #1223

chore: Update hierarchy query library #1223

Conversation

bprusinowski commented Oct 17, 2023 • edited Loading

vercel bot commented Oct 17, 2023 • edited Loading

Rdataflow commented Oct 17, 2023

bprusinowski commented Oct 18, 2023

Rdataflow commented Oct 18, 2023

Rdataflow commented Oct 18, 2023

bprusinowski commented Oct 19, 2023 • edited Loading

Rdataflow commented Oct 19, 2023

Rdataflow commented Nov 2, 2023

bprusinowski commented Nov 2, 2023

Rdataflow commented Nov 3, 2023

bprusinowski commented Nov 3, 2023

Rdataflow commented Nov 3, 2023

bprusinowski commented Nov 8, 2023

bprusinowski commented Feb 23, 2024

bprusinowski commented Oct 17, 2023 •

edited

Loading

vercel bot commented Oct 17, 2023 •

edited

Loading

bprusinowski commented Oct 19, 2023 •

edited

Loading