Operating real self-hosted runners behind an authenticated control plane

GitHub Actions already schedules jobs, evaluates workflow syntax, and records run history. A separate control plane should not pretend to replace those responsibilities. Its useful job is narrower: make the operating state of self-hosted runners legible, apply repository and lane policy consistently, preserve evidence, and show when its own view may be stale.

Runnerly now performs that job in a live CaeliCode staging environment. The runners are real organization-scoped self-hosted runners. The dashboard is real and available after GitHub authentication. The state shown in this article is not a fictional product screen or a proposed customer deployment.

That distinction matters because a polished diagram can easily outrun an implementation. The evidence here is deliberately divided into three access levels. The public health endpoint confirms that the control-plane process is responding. The public authentication-status endpoint confirms that GitHub authentication is required. The Runnerly root leads into that authentication boundary, while administrative data remains unavailable to anonymous requests. The implementation repository is private and therefore supports internal review, not independent public inspection.

What the authenticated review established

A timestamped authenticated review on August 18, 2026 showed live runner records, current status and labels, repository and workflow inventory, recent GitHub events, policy evaluation, evidence exports, audit activity, and backup controls. Dynamic totals and private identifiers are intentionally omitted. The relevant fact is not a particular fleet size; it is that the view was populated from operating runners and current organization data.

The review also showed the GitHub App installation and reconciliation path ready, recent webhook processing, and live browser updates. Those observations establish a working control plane. They do not establish customer use, external adoption, service-level performance, high availability, regulatory certification, or an ability to manage arbitrary organizations.

There is a second important boundary. Some capabilities are implemented in code but were not exercised end to end during this review. They include issuing a GitHub runner registration token through the App integration and using the optional host agent for machine-level telemetry. Those capabilities should be described as implemented, not as demonstrated operating outcomes.

Everything beyond that line remains design work: complete runner create, rotate, drain, and remove lifecycle; signed per-agent enrollment; multi-tenant isolation; a highly available data tier; and cryptographically tamper-evident evidence storage. Being explicit about those limits makes the live portion more credible, not less.

One state view from three imperfect signals

A control plane that relies only on webhooks will eventually be wrong. Deliveries can be delayed, retried, or missed. A control plane that relies only on polling will be slow and expensive. Runnerly therefore uses three complementary paths.

GitHub webhooks provide fast repository, workflow-job, and workflow-run events. GitHub documents the event vocabulary and payload boundaries. In the observed deployment, accepted events update normalized operational records and are pushed to the signed-in browser over server-sent events.

A frequent organization-runner refresh asks GitHub for current runner status, busy state, labels, and group information. The relevant self-hosted runner API and registration-token behavior answer a different question from a workflow event: what does GitHub currently believe about each registered runner?

A slower full reconciliation refreshes runners, groups, repositories, and related state. Its purpose is correction. If an event was missed or a previous observation became stale, reconciliation rebuilds the control-plane view from GitHub rather than allowing the error to persist indefinitely.

The current configuration uses a roughly 30-second lightweight runner refresh and a five-minute full reconciliation interval. Those are implementation settings, not availability guarantees. They define the expected freshness envelope under normal operation; alerts and timestamps are still needed to reveal when either loop stops succeeding.

Open full size ↗

GitHub webhook events, a frequent organization-runner heartbeat, and a slower full reconciliation flow into the Runnerly control plane. The control plane stores normalized state in SQLite, evaluates policy, records evidence, and streams updates to a GitHub-authenticated dashboard. An optional host agent is shown with a dashed line because it is implemented but not required for the observed dashboard state.

This model also explains why the dashboard can show both live state and confidence in that state. A runner marked online is useful; the time of the last successful heartbeat, reconciliation, or webhook is what tells an operator whether the assertion is still trustworthy.

Authentication is part of the operating architecture

Only process health and authentication status are intentionally public. Runner inventory, repository policy, jobs, audit data, exports, and backup operations require an authorized session.

The live path uses GitHub OAuth, requests organization-read access, verifies current organization membership and configured authorization conditions, and then issues an application session cookie. GitHub explains the meaning of the read:org scope. The implementation uses an HMAC-signed cookie with HttpOnly, SameSite, and production Secure attributes and a bounded session lifetime.

This is more precise than calling the system “SSO/OIDC.” GitHub OAuth proves the signed-in GitHub identity and supplies organization membership data; Runnerly makes its own authorization decision and manages its own session. That is the live mechanism, and it should be named accurately.

Webhook authentication is separate. GitHub recommends validating the delivery signature with the configured secret and a timing-safe comparison. The staging dashboard reported the webhook secret ready. However, the present code accepts webhook verification when no secret is configured, a development-friendly default that should become fail-closed for any production posture. A configuration error should disable event ingestion loudly, not silently remove its trust boundary.

Policy starts with repository visibility

Self-hosted runners execute repository-controlled code inside an environment an organization must operate and protect. Public repositories add an especially sharp risk because pull requests and forks can introduce untrusted workflow changes. GitHub's own guidance warns about granting public repositories access to self-hosted runner groups and documents the available controls.

Runnerly makes that boundary visible. Public repositories are treated as telemetry-only and expected to remain on GitHub-hosted execution. A public workflow targeting a self-hosted lane is recorded as a policy problem rather than being normalized as ordinary capacity. Private repositories can be assigned to a more specific lane and restricted runner group. A broad, generic self-hosted label is surfaced as a warning because it weakens the routing decision.

Open full size ↗

A workflow begins with repository visibility. Public repositories are routed to GitHub-hosted execution and a self-hosted label becomes a policy violation. Private repositories may proceed only through an explicit lane and restricted runner group; a broad self-hosted label produces a warning rather than silently expanding access.

This policy view does not itself prove that GitHub will reject every unsafe job. Enforcement ultimately depends on GitHub repository settings, runner-group access, workflow configuration, and the permissions of each integration. Runnerly's contribution is to reconcile those inputs, make violations inspectable, and keep public-repository behavior out of the private self-hosted fleet model.

The distinction is important in public engineering work. “A dashboard showed no violations at one timestamp” is a defensible observation. “The system guarantees no public code can ever run on a self-hosted machine” would require adversarial testing and hard enforcement evidence that this review did not establish.

From transient events to reviewable evidence

An operator needs more than a green runner badge. Useful evidence connects a workflow job to repository visibility, requested labels, the runner that accepted it, queue and execution timestamps, and the policy decision in force at that moment.

Runnerly records normalized job, repository, runner, event, and policy state in a local SQLite database. The authenticated dashboard exposes filtered operational views and can export evidence as JSON or CSV. The implementation also records application audit events and produces retained SQLite backups with restore tooling.

These are meaningful operating controls, but the wording must remain exact. The audit record is append-only through normal application paths; it is not cryptographically immutable. Backups located on the same host improve recovery from some application failures, but they are not offsite durability or disaster recovery. A restore command existing is not the same as a measured recovery objective.

The same discipline applies to the optional host agent. It can collect selected machine and service telemetry, but its present heartbeat uses a shared token. Signed per-agent enrollment, rotation, and revocation remain future security work. The authenticated dashboard's runner state does not depend on presenting that unfinished enrollment model as live.

Failure modes and present limitations

The staging deployment has several deliberate constraints:

• It is a single control-plane host with local SQLite, not a highly available or multi-tenant service.
• The public health endpoint reports process liveness; it does not prove GitHub API reachability, webhook freshness, database writability, or backup success.
• GitHub API and webhook state can be incomplete between correction cycles, so every operational status needs a timestamp.
• Runner-group metadata can be absent or ambiguous in upstream responses and should be displayed as unknown rather than inferred.
• Local backups do not establish offsite resilience.
• The optional agent's shared credential is not a finished enrollment security model.
• Webhook verification should be made mandatory outside explicit development mode.
• The private implementation cannot serve as independently reproducible public source evidence.

There is also no basis here for customer, compliance, uptime, throughput, or scale claims. “Designed for small, security-conscious environments” is accurate positioning. “Enterprise-grade,” “regulated-customer proven,” or “certified” would turn architectural intent into unsupported marketing.

Decision record: what Runnerly is now

Runnerly should be presented as a live authenticated staging control plane for real CaeliCode self-hosted runners. GitHub remains the workflow scheduler and source of authoritative organization state. Runnerly supplies a reconciled operator view, explicit repository and lane policy, evidence exports, audit records, and recovery-oriented backups.

The public product page and future writing should use three labels consistently:

1. Live and authenticated: runner visibility, GitHub App reconciliation, webhook events, frequent state refresh, server-sent browser updates, policy inspection, evidence exports, audit activity, and backups.
2. Implemented but not demonstrated in this review: registration-token and installer paths, plus the optional host agent.
3. Planned: full lifecycle control, signed enrollment, multi-tenancy, high availability, and stronger tamper evidence.

That vocabulary gives a technical reader enough information to evaluate the work without needing private repository access. It also prevents a future diagram from accidentally turning a staging system into a fictional commercial deployment.

What to carry into another control plane

The reusable lesson is not “build another dashboard.” It is to make uncertainty and trust boundaries visible:

• Combine fast events with periodic correction rather than trusting either one alone.
• Protect fleet and repository data behind identity plus explicit organization authorization.
• Treat public-repository access as a first-class execution boundary.
• Store enough job and policy context to explain why work ran where it did.
• Timestamp every live assertion and alert when the evidence source becomes stale.
• Separate process liveness from dependency readiness.
• Describe local audit and backup controls without upgrading them into immutability or disaster recovery.
• Keep live, implemented, and planned capabilities visually and verbally distinct.

Runnerly's strongest current evidence is modest but concrete: an authenticated operating surface connected to real runners, built around reconciliation and explicit policy. That is more useful than a larger claim that cannot be inspected.