GoHN — Hacker News API Wrapper for Go

GoHN is a wrapper for the Hacker News API for Go, inspired by the excellent go-github library.

It facilitates the use of the API by providing a simple interface to the API endpoints.

Features 🚀

• Get the top/new/best/ask/show/job stories
• High-performance comment retrieval with configurable rate limiting and worker pools
• Retrieve all comments (with metadata) for a story using optimized concurrent processing
• Up to 100x faster comment fetching for large threads with performance optimizations
• Retrieve the comments ordered as they appear in the story on the website
• Apply filters to retrieved items (stories, comments)
• Configurable HTTP client with connection pooling and custom timeouts

Usage 💻

Refer to the GoDoc for the full API reference.

Quick Start

Basic Usage

    // Instantiate a new client to retrieve data from the Hacker News API
    hn, err := gohn.NewClient(nil)
    if err != nil {
        panic(err)
    }

    // Use background context
    ctx := context.Background()

    // Get the top 500 stories' IDs
    topStoriesIds, _ := hn.Stories.GetIDs(ctx, gohn.TopStory)

    var story *gohn.Item
    // Retrieve the details of the first one
    if len(topStoriesIds) > 0 && topStoriesIds[0] != nil {
        story, _ = hn.Items.Get(ctx, *topStoriesIds[0])
    }

    if story == nil {
        panic("No story found")
    }

    // Print the story's title
    fmt.Println("Title:", *story.Title)

    // Print the story's author
    fmt.Println("Author:", *story.By)

    // Print the story's score
    fmt.Println("Score:", *story.Score)

    // Print the story's URL
    fmt.Println("URL:", *story.URL)

    fmt.Println()
    fmt.Println()

    if story.Kids == nil {
        fmt.Println("No comments found")
        return
    }

    // Retrieve all the comments for that story
    // UnescapeHTML is applied to each retrieved item to unescape HTML characters

    commentsMap, err := hn.Items.FetchAllDescendants(ctx, story, processors.UnescapeHTML())

    if err != nil {
        panic(err)
    }
    if len(commentsMap) == 0 {
        fmt.Println("No comments found")
        return
    }

    fmt.Printf("Comments found: %d\n", len(commentsMap))
    fmt.Println()

    // Create a Story struct to hold the story and its comments
    storyWithComments := gohn.Story{
        Parent:          story,
        CommentsByIdMap: commentsMap,
    }

    // Calculate the position of each comment in the story
    storyWithComments.SetCommentsPosition()

    // Get an ordered list of comments' IDs (ordered by position)
    orderedIDs, err := storyWithComments.GetOrderedCommentsIDs()

    if err != nil {
        panic(err)
    }

    // Print the comments
    for _, id := range orderedIDs {
        comment := commentsMap[id]
        if comment.Text != nil {
            fmt.Println(*comment.Text)
            fmt.Println()
        }
    }

Optimized Usage for Better Performance

For large comment threads (50+ comments), use the optimized client and functions:

    // Create optimized client with custom rate limiting
    hn, err := gohn.NewClientWithOptions(&gohn.ClientOptions{
        RateLimit: 200 * time.Millisecond, // 5 requests per second (be respectful)
        BurstSize: 10,                     // Allow bursts of 10 requests
    })
    if err != nil {
        panic(err)
    }

    ctx := context.Background()

    // Get story (same as before)
    story, _ := hn.Items.Get(ctx, storyID)

    // Use optimized function for faster comment retrieval
    commentsMap, err := hn.Items.FetchAllDescendantsOptimized(
        ctx, 
        story, 
        processors.UnescapeHTML(), 
        0, // Auto-determine optimal worker count
    )

    if err != nil {
        panic(err)
    }
    
    fmt.Printf("Comments fetched: %d\n", len(commentsMap))
    // Process comments same as before...

Performance Comparison

Scenario	Original	Optimized	Improvement
10 comments	9.0 seconds	0.09 seconds	100x faster
271 comments	271+ seconds	~49 seconds	5.5x faster

Note: Always be respectful of the Hacker News API. The default rate limit is 1 req/sec. For better performance, 5 req/sec (200ms) is recommended, with 10 req/sec as an absolute maximum.

Examples

• example/main.go - Basic usage example
• example/quick_performance.go - Performance testing example

For detailed performance information, see PERFORMANCE_IMPROVEMENTS.md.

Semantic Versioning 🥚

As this library is not yet in version 1.0.0, the API may have breaking changes between minor versions.

Contributing 🤝🏼

Feel free to fork this repo and create a PR. I will review them and merge, if ok.

License 📝

MIT