Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automatically Describe DFs Going Into Data Explorer #24

Open
emeeks opened this issue Aug 2, 2018 · 4 comments
Open

Automatically Describe DFs Going Into Data Explorer #24

emeeks opened this issue Aug 2, 2018 · 4 comments
Labels
enhancement New feature or request

Comments

@emeeks
Copy link
Member

emeeks commented Aug 2, 2018

df.describe(include="all") should run and be included as metadata for any dataframe that's being sent to the Data Explorer component.

@rgbkrk
Copy link
Member

rgbkrk commented Aug 2, 2018

Fantastic idea!

screen shot 2018-08-02 at 11 57 07 am

I think we can include it as part of the fields object (🎩 tip to @emeeks for suggesting this in person). It could come out like this:

screen shot 2018-08-02 at 11 38 22 am

We need to impact:

@alexandercbooth and I prototyped a version that captures the summary statistics just now and this is the code we came up with for pandas:

diff --git a/pandas/io/json/table_schema.py b/pandas/io/json/table_schema.py
index 2dc176648..0460868c1 100644
--- a/pandas/io/json/table_schema.py
+++ b/pandas/io/json/table_schema.py
@@ -113,6 +113,10 @@ def convert_pandas_type_to_json_field(arr, dtype=None):
             field['tz'] = arr.dt.tz.zone
         else:
             field['tz'] = arr.tz.zone
+
+    # TODO: get this to be part of the spec for https://frictionlessdata.io/specs/table-schema/
+    if hasattr(arr, 'describe'):
+        field['summary'] = arr.describe(include="all").to_dict()
     return field

Admittedly, I don't know what the performance implications are. 😬 Perhaps this is fine if it's already being serialized.

Notebook that uses this and will be useful for debugging: https://gist.github.com/rgbkrk/e1b477641128213db71efa34cfdbb8a7

cc @TomAugspurger

@alexandercbooth wants to take on bringing this into pandas.

@stale
Copy link

stale bot commented Aug 5, 2019

This issue hasn't had any activity on it in the last 90 days. Unfortunately we don't get around to dealing with every issue that is opened. Instead of leaving issues open we're seeking to be transparent by closing issues that aren't being prioritized. If no other activity happens on this issue in one week, it will be closed.
It's more than likely that just by me posting about this, the maintainers will take a closer look at these long forgotten issues to help evaluate what to do next.
If you would like to see this issue get prioritized over others, there are multiple avenues 🗓:

  • Ask how you can help with this issue 👩🏿‍💻👨🏻‍💻
  • Help solve other issues the team is currently working on 👨🏾‍💻👩🏼‍💻
  • Donate to nteract so we can support developers to work on these features and bugs more regularly 💰🕐

Thank you!

@hydrosquall
Copy link
Member

A related project for ideas around what sorts of summary statistics could be piped into the table:

https://github.com/pandas-profiling/pandas-profiling

@captainsafia
Copy link
Member

For Hacktoberfest 2019 participants: resolving this issue will require changes across the pandas and nteract repos.

@rgbkrk's comment above is a great place to start on the changes required on the Pythons side -- which is the first place to start with this modification.

@captainsafia captainsafia transferred this issue from nteract/nteract May 6, 2020
@willingc willingc added the enhancement New feature or request label Jun 4, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

5 participants