Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RFC: Creating SIG Models #314

Merged
merged 2 commits into from
Nov 12, 2020
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
91 changes: 91 additions & 0 deletions rfcs/20201023-sig-models.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
# Creating SIG Models

| Status | Proposed |
:-------------- |:---------------------------------------------------- |
| **RFC #** | 314 |
| **Author(s)** | Jaeyoun Kim ([email protected]), Jing Li ([email protected]), Mike Liang ([email protected]), Shuangfeng Li ([email protected]) |
| **Sponsor** | Thea Lamkin ([email protected]) |
| **Updated** | 2020-10-30 |

## What is this group for?

This group is for discussions and collaborations on enabling community
contributions to [TensorFlow Model Garden](https://github.com/tensorflow/models)
and [Tensorflow Hub](https://github.com/tensorflow/hub).

SIG Models will focus on empowering the community to contribute state-of-the-art
model implementation in TensorFlow 2. It will benefit the whole community by
providing recommended implementations and models with reproducible results.

SIG Models will have several subgroups (e.g., SIG Models Vision and SIG Models
NLP) covering different machine learning areas. There are several SIG leads for
each group to coordinate the contributions, run community events like contests,
and maintain the code quality through the review process. Each group has the
flexibility to operate differently.

SIG Models is also a place for community discussions and sharing best practices of
using TensorFlow 2 for state-of-the-art research. Furthermore, SIG Models provides
product feedback to help TensorFlow to be improved.

## Who will be part of it?
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about keras-cv/keras-nlp?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same question.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bhack @AakashKumarNain Yes, the Keras team (keras-cv/nlp) will contribute to the SIG Models.


SIG Models are open, membership will be entirely public, and anybody interested
in model contributions or participating in the discussion can join the SIG.
There can be different ways to participate:

* Everyone in the community can pick GitHub model tasks and contribute models.
* They can also join the regular SIG meetings or email groups to participate in
the discussions. To participate, request an invitation to join the SIG mailing
list. Archives of the mailing list are publicly accessible.

We will identify SIG leads from the community to run the SIG Models groups.
Initially, we plan to set up the SIG Models Vision group:

* SIG Models Vision leads
* George Thiruvathukal, [email protected] (Loyola University Chicago)
* Yung-Hsiang Lu, [email protected] (Purdue University)
* Co-leads and sponsors from TensorFlow
* Jaeyoun Kim, [email protected] (TensorFlow Model Garden)
* Jing Li, [email protected] (TensorFlow Model Garden)
* Mike Liang, [email protected] (TensorFlow Hub)
* Shuangfeng Li, [email protected] (TensorFlow Lite)

## What initial problems will the group tackle?

We will start with the SIG Models Vision group. Initially, the group leads will
run a contest to motivate the community to contribute state-of-the-art computer
vision models using TensorFlow 2.

For example:

* Define/release model implementation tasks, rules, and evaluation criteria for
the contest
* Review initial submissions and provide more support for selected high-quality
submissions
* Review final submissions and announce results
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is the lifecycle of such models? I can see at least: draft, initial, final
Any guarantees of maintainability? Can models be deprecated over time?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are also trying to figure if a bot could do some keep-alive/MIA checks tensorflow/addons#2024

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The real question is also how to implement something like tensorflow/addons#236. I don't know if this new SIG will have the resource to implement something like to solve this open topic. I think it would be really useful for all the SIGs but also for the TF ecosystem.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lc0 The lifecycle of community models will be one of the topics the SIG Models will discuss. We expect that we will also have a periodic review for models. Thank you for your comment.

@bhack Thank you for your suggestion. I agree with you that it would be useful to have such an analyzer. We will definitely discuss it once the TF SIG Models is created. We will ask for more inputs from you.


## What modes of communication do you intend to use?
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think would be helpful to also define what is expected use-case.

Do we see it as python module, that can be imported downstream, hence models are tested and maintained.
Or just a collection of scripts, that contributed by different people at different times?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment is connected to the models's reusable components topic.
Some times ago we had a ticket about this at #223

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are many dups or semi-dups like tensorflow/addons#1903 (comment) that confuse users and third party SIG maintainers.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lc0 @bhank Thank you for your suggestions and comments. We will add that as one of the first topics for open discussion after the SIG Models created. We want to start with a small scope and expand it later based on community inputs.


We plan to have a mailing list
([[email protected]](https://groups.google.com/a/tensorflow.org/forum/#!forum/models)),
arrange a regular (e.g., monthly) video conference call rotated by the SIG
leads. We will also use a Gitter chat channel
(https://gitter.im/tensorflow/sig-models) for discussions.

As we have multiple SIG subgroups, each subgroup may run meetings and events
using different modes of communication.

## Launch plan

* Video conference calls with initial interested parties to finalize the charter
* SIG set up with initial group members
* SIG added to community pages on tensorflow.org
* SIG leads and sponsors start discussions about initial work items
* Write a blog post about the SIG Models with the initial achievements and
welcome more members

## Code of Conduct

As with all forums and spaces related to TensorFlow, SIG Models is subject to
the [TensorFlow Code of
Conduct](https://github.com/tensorflow/tensorflow/blob/master/CODE_OF_CONDUCT.md).