starcat-trending-api

Starcat Trending API

中文说明

Preferred install method:

brew tap starcat-app/starcat
brew trust starcat-app/starcat
brew install --cask starcat

Useful links:

• Home and downloads: https://starcat.ink
• Mac App Store: search for Starcat for GitHub
• Public support and release notes: https://github.com/starcat-app/starcat-pro
• Starcat App Homebrew tap: https://github.com/starcat-app/homebrew-starcat
• CLI / MCP: starcat-cli / Homebrew tap
• AI Agent Skill: https://github.com/starcat-app/starcat-skill
• Browser plugins: Chrome / Safari
• Launcher integrations: Alfred / uTools / Raycast
• Documentation: https://github.com/starcat-app/starcat-docs
• Website source: https://github.com/starcat-app/starcat-site
• Localization: https://github.com/starcat-app/starcat-localization

Self-hostable support APIs:

• starcat-sharing-api
• starcat-trending-api
• starcat-weekly-api
• starcat-wiki-api
• starcat-recommend-api
• starcat-discovery-api

Starcat provides hosted defaults for normal users. This API is open source so advanced users can inspect it, run it locally, or deploy their own instance.

GitHub Trending data API written in Go with a consistent response envelope.

trending-api uses GitHub as its only source. starcat-weekly-api provides zread weekly rankings through GET /api/v1/trending/zread; this service does not expose them.

Features

• Three-layer architecture: spider (HTML crawler) → store (SQLite) → enricher (GitHub API metadata)
• Bearer Token authentication: required for every /api/v1/* and /internal/* endpoint
• Token Pool: multiple redundant GitHub PATs with quota-aware selection and failover
• Rate Limit backoff: reads the X-RateLimit-Remaining header and slows down when quota is low
• Priority queue: enriches top-ranked entries first (enrich_priority DESC)
• Admin endpoints: manually trigger synchronization through /internal/sync/*

Quick Start

Requirements

• Go 1.25+

Run Locally

cp .env.example .env
# Edit .env and set API_KEYS and GITHUB_TOKENS
cd starcat-trending-api
go run ./cmd/server/

The default port is 5002.

.env Configuration

API Endpoints

Every data endpoint requires an Authorization: Bearer <api-key> header.

GET /api/v1/ping (Authentication Required)

Returns the service identity and the build version injected from the release tag:

{"schema_version":1,"data":{"service":"trending","version":"1.2.3","ok":true}}

GET /api/v1/repos?lang=&since=&limit= (Authentication Required)

Returns a list of trending repositories with fields enriched through the GitHub API.

Note: The endpoint does not accept a source=* parameter. trending-api uses GitHub as its only source. For zread data, use weekly-api GET /api/v1/trending/zread.

See StarcatRepoCardDTO in internal/model/repo_card.go for the response format.

GET /api/v1/languages (Authentication Required)

Returns a language list with repository counts, aggregated from the trending_repos table at request time. The list contains only languages with current repositories plus one __uncategorized__ entry for repositories whose language is NULL or empty.

Before 2026-06-11, v1 returned the complete language menu scraped from GitHub Trending (more than 700 entries, most with no data in our database). The endpoint now aggregates languages from available data. The key and label response fields remain backward-compatible, and the response adds count. The client sidebar uses this endpoint to populate its trending language list.

Example response:

{
  "schema_version": 1,
  "data": [
    { "key": "Python", "label": "Python", "count": 42 },
    { "key": "Go", "label": "Go", "count": 31 },
    { "key": "TypeScript", "label": "TypeScript", "count": 18 },
    { "key": "__uncategorized__", "label": "Uncategorized", "count": 5 }
  ],
  "meta": {
    "total": 4,
    "generated_at": "2026-06-11T12:00:00Z",
    "cache_status": "fresh"
  }
}

Field descriptions:

• key: Stable language identifier using GitHub's normalized language name, such as Go or Python. Uncategorized entries always use __uncategorized__, which can be passed to GET /api/v1/repos?lang=__uncategorized__.
• label: Display name. Regular languages use key; uncategorized entries use Uncategorized. Clients may override the label through their own i18n.
• count: Number of available, enriched repositories for the language in the current trending_repos table, combined across all three periods.

Sort order: Uncategorized entries always appear last. Other languages use count DESC with key ASC as a stable tiebreaker.

Meaning of the __uncategorized__ Sentinel in /api/v1/repos

GET /api/v1/repos?lang=__uncategorized__ is equivalent to querying language IS NULL OR language = ''. It returns every trending repository for which GitHub did not identify a primary language and neither the spider nor the enricher could fill the value.

GET /api/v1/users?lang=&since=&sponsorable= (Authentication Required)

Returns a list of trending developers.

Admin Endpoints (Authentication Required)

GET /healthz (Public)

Health check that returns ok.

Operations and Metrics

• GET /internal/stats: period counts, visibility, enrichment backlog, unavailable rows, and freshness.
• GET /internal/metrics/{summary,timeseries,routes,status-codes}: authenticated aggregate request metrics.
• Existing GET /api/v1/repos and GET /api/v1/languages are the bounded Admin Console data views.

Metrics retain route templates only; credentials, queries, bodies, client addresses, and real path parameters are excluded.

Authentication

Every /api/v1/* and /internal/* endpoint requires an Authorization: Bearer <api-key> header.

Generate a new key:

bash ../scripts/gen-api-key.sh

Project Structure

starcat-trending-api/
├── cmd/server/main.go              # Entry point: wires the three layers, scheduler, and authentication
├── internal/
│   ├── spider/                     # HTML crawler (goquery)
│   ├── store/                      # SQLite persistence
│   ├── enricher/                   # GitHub API field enrichment and rate limiting
│   ├── tokenpool/                  # GitHub Token Pool
│   ├── scheduler/                  # cron scheduling
│   ├── handler/                    # HTTP handlers (v1 and admin)
│   ├── middleware/                 # Bearer authentication middleware
│   └── model/                      # Data models (DB, DTO, and Envelope)
├── .env.example                    # Configuration template
├── fly.toml                        # Fly.io deployment configuration
├── Dockerfile
└── Makefile

Deployment (Fly.io)

fly secrets set \
  API_KEYS="sk-starcat-prodKey1,sk-starcat-prodKey2" \
  GITHUB_TOKENS="ghp_token1,ghp_token2,ghp_token3" \
  STORE_FILE="/data/trending.db" \
  METRICS_STORE_FILE="/data/trending-metrics.db" \
  -a starcat-trending-api

fly deploy -a starcat-trending-api

Technology

• net/http: Go standard library with no framework dependency
• goquery: HTML parsing with jQuery-like selectors
• SQLite: modernc.org/sqlite (pure Go, no CGO)
• cron: robfig/cron/v3 for scheduled jobs
• godotenv: loads environment variables from .env