Inspect & standardize identifiers#

To make data queryable by an entity identifier, one needs to ensure that identifiers comply to a chosen standard.

Bionty enables this by mapping metadata on the versioned ontologies using inspect().

For terms that are not directly mappable, we offer (also see Search & lookup terms):

import bionty as bt
import pandas as pd

Inspect and mapping synonyms of gene identifiers#

To illustrate it, let us generate a DataFrame that stores a number of gene identifiers, some of which corrupted.

data = {
    "gene symbol": ["A1CF", "A1BG", "FANCD1", "corrupted"],
    "ncbi id": ["29974", "1", "5133", "corrupted"],
    "ensembl_gene_id": [
        "ENSG00000148584",
        "ENSG00000121410",
        "ENSG00000188389",
        "ENSGcorrupted",
    ],
}
df_orig = pd.DataFrame(data).set_index("ensembl_gene_id")

df_orig
gene symbol ncbi id
ensembl_gene_id
ENSG00000148584 A1CF 29974
ENSG00000121410 A1BG 1
ENSG00000188389 FANCD1 5133
ENSGcorrupted corrupted corrupted

First we can check whether any of our values are mappable against the ontology reference.

Tip: available fields are accessible via gene_bt.fields

gene_bt = bt.Gene()

gene_bt

Gene
Species: human
Source: ensembl, release-110
#terms: 77043

📖 Gene.df(): ontology reference table
🔎 Gene.lookup(): autocompletion of terms
🎯 Gene.search(): free text search of terms
🧐 Gene.inspect(): check if identifiers are mappable
👽 Gene.map_synonyms(): map synonyms to standardized names
🪜 Gene.diff(): difference between two versions
🔗 Gene.ontology: Pronto.Ontology object

gene_bt.inspect(df_orig.index, gene_bt.ensembl_gene_id)

✅ 3 terms (75.0%) are mapped

🔶 1 terms (25.0%) are not mapped

{'mapped': ['ENSG00000148584', 'ENSG00000121410', 'ENSG00000188389'],
 'not_mapped': ['ENSGcorrupted']}

The same procedure is available for ncbi_gene_id or gene symbol. First, we inspect which symbols are mappable against the ontology.

gene_bt.inspect(df_orig["ncbi id"], gene_bt.ncbi_gene_id)

✅ 3 terms (75.0%) are mapped

🔶 1 terms (25.0%) are not mapped

{'mapped': ['29974', '1', '5133'], 'not_mapped': ['corrupted']}

gene_bt.inspect(df_orig["gene symbol"], gene_bt.symbol)

🔶 The identifiers contain synonyms!
   To increase mappability, standardize them via '.map_synonyms()'

✅ 2 terms (50.0%) are mapped

🔶 2 terms (50.0%) are not mapped

{'mapped': ['A1CF', 'A1BG'], 'not_mapped': ['FANCD1', 'corrupted']}

Apparently 2 of the gene symbols are mappable. Bionty further warns us that some of our symbols can be mapped into standardized symbols.

Mapping synonyms returns a list of standardized terms:

mapped_symbol_synonyms = gene_bt.map_synonyms(df_orig["gene symbol"])

mapped_symbol_synonyms

['A1CF', 'A1BG', 'BRCA2', 'corrupted']

Optionally, only returns a mapper of {synonym : standardized name}:

gene_bt.map_synonyms(df_orig["gene symbol"], return_mapper=True)

{'FANCD1': 'BRCA2'}

We can use the standardized symbols as the new index:

df_curated = df_orig.reset_index()
df_curated.index = mapped_symbol_synonyms

df_curated
ensembl_gene_id gene symbol ncbi id
A1CF ENSG00000148584 A1CF 29974
A1BG ENSG00000121410 A1BG 1
BRCA2 ENSG00000188389 FANCD1 5133
corrupted ENSGcorrupted corrupted corrupted

You may return a DataFrame with a boolean column indicating if the identifiers are mappable:

gene_bt.inspect(df_curated.index, gene_bt.symbol, return_df=True)

✅ 3 terms (75.0%) are mapped

🔶 1 terms (25.0%) are not mapped
__mapped__
A1CF True
A1BG True
BRCA2 True
corrupted False

Standardize and look up unmapped CellMarker identifiers#

Depending on how the data was collected and which terminology was used, it is not always possible to curate values. Some values might have used a different standard or be corrupted.

This section will demonstrate how to look up unmatched terms and curate them using CellMarker.

First, we take an example DataFrame whose index containing a valid & invalid cell markers (antibody targets) and an additional feature (time) from a flow cytometry dataset.

markers = pd.DataFrame(
    index=[
        "KI67",
        "CCR7",
        "CD14",
        "CD8",
        "CD45RA",
        "CD4",
        "CD3",
        "CD127a",
        "PD1",
        "Invalid-1",
        "Invalid-2",
        "CD66b",
        "Siglec8",
        "Time",
    ]
)

Let’s instantiate the CellMarker ontology with the default database and version.

cellmarker_bt = bt.CellMarker()

cellmarker_bt





CellMarker
Species: human
Source: cellmarker, 2.0
#terms: 15466

📖 CellMarker.df(): ontology reference table
🔎 CellMarker.lookup(): autocompletion of terms
🎯 CellMarker.search(): free text search of terms
🧐 CellMarker.inspect(): check if identifiers are mappable
👽 CellMarker.map_synonyms(): map synonyms to standardized names
🪜 CellMarker.diff(): difference between two versions
🔗 CellMarker.ontology: Pronto.Ontology object









Now let’s check which cell markers from the file can be found in the reference:






cellmarker_bt.inspect(markers.index, cellmarker_bt.name)









🔶 Detected inconsistent casing of mapped terms!
   For best practice, standardize casing via '.map_synonyms()'





✅ 10 terms (71.4%) are mapped





🔶 4 terms (28.6%) are not mapped





{'mapped': ['KI67',
  'CCR7',
  'CD14',
  'CD8',
  'CD45RA',
  'CD4',
  'CD3',
  'PD1',
  'CD66b',
  'Siglec8'],
 'not_mapped': ['CD127a', 'Invalid-1', 'Invalid-2', 'Time']}









Logging suggests we map synonyms:






synonyms_mapper = cellmarker_bt.map_synonyms(markers.index, return_mapper=True)









Now we mapped 2 additional terms:






synonyms_mapper









{'KI67': 'Ki67', 'CCR7': 'Ccr7', 'CD14': 'Cd14', 'CD4': 'Cd4'}









Let’s replace the synonyms with standardized names in the markers DataFrame:






markers.rename(index=synonyms_mapper, inplace=True)









From the logging, it can be seen that 4 terms were not found in the reference!


Among them Time, Invalid-1 and Invalid-2 are non-marker channels which won’t be curated by cell marker.






cellmarker_bt.inspect(markers.index, cellmarker_bt.name)









✅ 10 terms (71.4%) are mapped





🔶 4 terms (28.6%) are not mapped





{'mapped': ['Ki67',
  'Ccr7',
  'Cd14',
  'CD8',
  'CD45RA',
  'Cd4',
  'CD3',
  'PD1',
  'CD66b',
  'Siglec8'],
 'not_mapped': ['CD127a', 'Invalid-1', 'Invalid-2', 'Time']}









We don’t really find CD127a, let’s check in the lookup with auto-completion:






lookup = cellmarker_bt.lookup()













lookup.cd127









CellMarker(name='CD127', synonyms='', gene_symbol='IL7R', ncbi_gene_id='3575', uniprotkb_id='P16871', _5='cd127')









Indeed we find it should be cd127, we had a typo there with cd127a.


Now let’s fix the markers so all of them can be linked:




Tip


Using the .lookup instead of passing a string helps eliminate possible typos!








curated_df = markers.rename(index={"CD127a": lookup.cd127.name})









Optionally, run a fuzzy match:






cellmarker_bt.search("CD127a").head()













  
    

      
      synonyms
      gene_symbol
      ncbi_gene_id
      uniprotkb_id
      __agg__
      __ratio__
    

    

      name
      
      
      
      
      
      
    

  
  
    

      CD127
      
      IL7R
      3575
      P16871
      cd127
      90.909091
    

    

      CD167a
      
      None
      None
      None
      cd167a
      83.333333
    

    

      CD172a
      
      None
      None
      None
      cd172a
      83.333333
    

    

      CD121a
      
      None
      None
      None
      cd121a
      83.333333
    

    

      CD120a
      
      TNFRSF1A
      7132
      P19438
      cd120a
      83.333333
    

  







OK, now we can try to run curate again and all cell markers are linked!






cellmarker_bt.inspect(curated_df.index, cellmarker_bt.name)









✅ 11 terms (78.6%) are mapped





🔶 3 terms (21.4%) are not mapped





{'mapped': ['Ki67',
  'Ccr7',
  'Cd14',
  'CD8',
  'CD45RA',
  'Cd4',
  'CD3',
  'CD127',
  'PD1',
  'CD66b',
  'Siglec8'],
 'not_mapped': ['Invalid-1', 'Invalid-2', 'Time']}