Skip to main content

​
Overview

NuGet.Jobs.GitHubIndexer is a scheduled background job that builds a picture of which NuGet packages are being used by popular open-source C# projects on GitHub. It queries the GitHub Search API for C# repositories exceeding a configurable star threshold (default: 100 stars), clones only the default branch of each repository using LibGit2Sharp, and scans the tree for dependency declaration files (packages.config, *.csproj, *.props, *.targets). Extracted package IDs are deduplicated and serialized as a JSON blob named GitHubUsage.v1.json in the content Azure Blob Storage container. The job is designed as a one-shot console application run by a scheduler (deployed as a Windows service via NSSM). On each run it fetches a fresh list of repositories from GitHub, processes them in parallel across a configurable thread pool (default: 32 threads), and uploads the result before exiting. A disk-based cache (DiskRepositoriesCache) persists each repository’s parsed dependency list to the local temp directory during a run so that if processing is interrupted or a repository times out, already-processed results are not lost within that run. A key design constraint is the GitHub Search API limit of 1,000 results per query. To retrieve more repositories than this cap allows, GitHubSearcher performs sliding-window pagination: it issues multiple pages of results ordered by descending star count and advances the upper star bound to the lowest star count of the last batch, repeating until all repositories above the minimum star threshold have been enumerated.

​
Role in System

GitHub Discovery

Uses the Octokit IGitHubClient to search for C# repositories by star count, paginating through results to work around the 1,000-result-per-query API cap. Rate limit headers are inspected to implement automatic throttle-aware delays.

Sparse Git Clone

Rather than downloading a full repository archive, FetchedRepo initializes a local bare repository with LibGit2Sharp, fetches only the default branch ref, and uses CheckoutPaths to materialize only the specific dependency files identified by Filters.

Dependency Extraction

ConfigFileParser and RepoUtils parse both packages.config (via NuGet.Packaging.PackagesConfigReader) and SDK-style project files (via XmlDocument scanning for PackageReference nodes). Files over 1 MB are skipped.

Usage Blob Output

Produces GitHubUsage.v1.json in the content blob container — a JSON array of repository records sorted by descending star count, each listing its NuGet package dependencies. Only repositories with at least one detected dependency are included.

​
Key Files and Classes

​
Dependencies

​
NuGet Package References

​
Internal Project References

​
Notable Patterns and Implementation Details

The job runs repository indexing on dedicated Thread objects rather than Task.Run. LibGit2Sharp operations are inherently synchronous and CPU/IO-bound. Spawning threads from the background thread pool avoids starving the async Task scheduler. Each thread is marked IsBackground = true so the process can exit even if a thread is still running after a cancellation.
If any repository exceeds RepoIndexingTimeout (default: 150 minutes), the controlling SemaphoreSlim.WaitAsync times out, which propagates a cancellation to all parallel workers and crashes the process. The intent is to prevent a single hung git operation from blocking the job indefinitely; the process will restart and pick up from the cached results.
GitHubSearcher uses a sliding upper-star-bound strategy to work around the GitHub Search API’s hard cap of 1,000 results per query. After each full page set, it sets the next query’s upper star bound to the lowest star count observed so far, effectively partitioning the result space by star count ranges. A warning is logged when the last page of a range shows uniform star counts, which would cause data loss because the partition cannot be further subdivided.
The GitHub Search API can return duplicate repository entries across pages when the result set shifts between requests. GitHubSearcher.GetPopularRepositories deduplicates by grouping on the repository’s full name (the Id field) and taking the first occurrence.
GitHubSearchWrapper applies a forcible double-timeout (GitHubRequestTimeout * 2) on top of the Octokit/HttpClient built-in timeout via TaskExtensions.ExecuteWithTimeoutAsync. This guards against cases where the underlying HTTP request hangs without triggering the built-in timeout, which has been observed in production.
Files larger than 1 MB (ReposIndexer.MaxBlobSizeBytes = 1 << 20) are skipped during checkout with a warning. This prevents excessively large auto-generated project files from consuming disproportionate processing time or memory.
The local working directory used for repository clones and the disk cache is %TEMP%\NuGet.Jobs.GitHubIndexer. Both the repos and cache subdirectories are created at the start of each run and deleted on successful completion. Read-only files (common in .git folders) are explicitly made writable before deletion to work around Directory.Delete limitations on Windows.