Add cellxgene census component for data intake into openpipeline

[ddemaeyer]

With the new release of cellxgene-census an api for the querying and the generation of h5ads from online data is available. The goal is to add a component to convert the data to an h5mu which we can process in openpipeline.

Example tutorial of using cellxgene-census code from: https://chanzuckerberg.github.io/cellxgene-census/notebooks/api_demo/census_datasets.html

import cellxgene_census

## Querying

with cellxgene_census.open_soma() as census:
    
    # Reads SOMADataFrame as a slice
    cell_metadata = census["census_data"]["homo_sapiens"].obs.read(
        value_filter = "sex == 'female' and cell_type in ['microglial cell', 'neuron']",
        column_names = ["assay", "cell_type", "tissue", "tissue_general", "suspension_type", "disease"]
    )
    
    # Concatenates results to pyarrow.Table
    cell_metadata = cell_metadata.concat()
    
    # Converts to pandas.DataFrame
    cell_metadata = cell_metadata.to_pandas()
 
    print(cell_metadata)

## Convert data into a h5ad

with cellxgene_census.open_soma() as census:
    adata = cellxgene_census.get_anndata(
        census = census,
        organism = "Homo sapiens",
        obs_value_filter = "cell_type in ['T cell']",
        column_names = {"obs": ["assay", "cell_type", "tissue", "tissue_general", "suspension_type", "disease"]},
    )
    
    print(adata)

Outstanding questions are

• What to do with naming of the vars: replace the soma_ids with the ensembl ids?
• What to do with the cell ids: currently there is no uniform cell_id set, only the soma Ids as integers

Let's discuss

[ddemaeyer]

After a discussion on 20230531 it was indicated that there is the need for a first component to perform the query to visualize and return the actual observations in the result. A second component would then return the full query as a h5mu.

Bringing this back to pipelines, the first component would be run prior to the second making it possible to run the component without executing the actual data generation.