Add comprehensive resources for exploring opensource LLMs and model weights on Huggingface

[Copilot]

This PR implements a complete workshop framework for exploring opensource Large Language Models (LLMs) on Huggingface Hub and understanding model weights usage.

Overview

The repository now provides multiple learning paths and practical examples for working with Huggingface models, from beginner-friendly quick starts to advanced weight management patterns.

What's Added

📓 Interactive Learning

• explore_llms.ipynb - Comprehensive Jupyter notebook with 20+ code cells covering:
  • Searching and filtering models on Huggingface Hub by task, popularity, and downloads
  • Loading model weights using multiple methods (AutoModel, custom configs, reduced precision)
  • Text generation with various models (GPT-2, BERT, DistilGPT-2)
  • Model architecture inspection and best practices

🐍 Executable Python Scripts

• simple_demo.py - Minimal working example for immediate text generation (perfect first experience)
• explore_models.py - CLI tool for model discovery, loading, and basic inference with detailed output
• model_weights_usage.py - Advanced patterns including:
  • Efficient model caching and downloading
  • Saving/loading models to/from local directories
  • Weight inspection with detailed statistics (min, max, mean, std dev)
  • Weight freezing for fine-tuning scenarios
  • Weight sharing between model instances
  • Different initialization strategies

📖 Documentation

• README.md - Comprehensive guide with installation, examples, model selection criteria, optimization tips, and resource links
• QUICKSTART.md - Quick start guide with immediate usage instructions, common use cases, and troubleshooting

🔧 Configuration

• requirements.txt - All necessary dependencies (transformers, torch, huggingface-hub, etc.)
• .gitignore - Excludes cache files, model weights, and build artifacts

Key Features

Model Exploration

from huggingface_hub import list_models, model_info

# Search for top text generation models
models = list(list_models(task="text-generation", sort="downloads", limit=10))

# Get detailed model information
info = model_info("gpt2")
print(f"Downloads: {info.downloads}, Task: {info.pipeline_tag}")

Weight Loading and Management

# Load with efficient caching
tokenizer = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("gpt2", torch_dtype=torch.float16)

# Inspect weights
for name, param in model.named_parameters():
    print(f"{name}: {param.shape}, {param.numel():,} parameters")

Practical Usage

# Simple text generation
from transformers import pipeline
generator = pipeline('text-generation', model='gpt2')
result = generator("Artificial intelligence is", max_length=50)

User Journey

• Beginners (5 minutes): python simple_demo.py → Read QUICKSTART.md
• Intermediate (30 minutes): python explore_models.py → Read README.md
• Advanced (1+ hour): python model_weights_usage.py → Complete explore_llms.ipynb

Examples Included

• Searching models by task and popularity
• Loading models in different precisions (fp32, fp16)
• Text generation with custom parameters
• Model size comparison (GPT-2 vs DistilGPT-2)
• Weight freezing for transfer learning
• Saving and loading models locally
• BERT masked language modeling
• Pipeline API for simplified inference

All code is fully documented with inline comments and docstrings. The examples use popular, lightweight models (GPT-2, DistilGPT-2, BERT) that are easy to download and run on standard hardware.

Original prompt

Explore opensource LLMs on Huggingface and model (weights) usage

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.