Skip to content

SkyPurza — Project Plan

Location: projects/skypilot/skypurza · Date: 2026-10-19 Model: Open-source broker implementing the SkyPilot open standard; closed-source value-added services (accounting, automatic dispatching, observability) for monetization. Reference architecture: kwedra-roko (pnpm monorepo: shared / service / infra-CDK / frontend / test packages, multi-region CDK pipeline with bake + DR stages, centralized naming.ts, docs/DESIGN.md).

1. Positioning

SkyPurza mirrors SkyPilot's proven open-core play: - OSS broker (Apache-2.0): full single-user/single-team experience — portable task YAML, multi-backend launch (AWS, Kubernetes, Slurm via login-node SSH), cost-optimized placement, spot failover, managed jobs. Interop with the SkyPilot open standard (task YAML schema, CLI verb parity, env-var contract) so existing SkyPilot YAMLs run unchanged. - Initial scope: AWS-only, multi-region — the optimizer places and fails over across AWS regions (cheapest region with availability, spot→on-demand fallback). Kubernetes and Slurm backends follow in later phases; the adapter interface is designed for them from day one. - Closed-source Platform services (monetized): exactly the gaps the SkyPilot community complains about in OSS: 1. Accounting — per-user/team usage ledger, TRES-style billing weights, chargeback/showback, settlement (pattern: kwedra-roko proxy-service/src/core/reservations.ts settlement logic). 2. Automatic dispatching — policy-driven placement across clusters/clouds: quotas, fairshare, priorities with aging, scheduler-initiated preemption, reservation-aware routing. 3. Observability — GPU health manager, goodput/utilization dashboards, Prometheus federation, per-job cost telemetry, alerting.

2. Repo layout (pnpm monorepo, kwedra-roko style)

skypurza/
├── pnpm-workspace.yaml
├── package.json
├── docs/DESIGN.md               # living design doc (kwedra-roko convention)
├── packages/
│   ├── oss/                     # ── OPEN SOURCE (Apache-2.0, public export) ──
│   │   ├── shared/              # types, task-YAML schema (SkyPilot standard), naming.ts
│   │   ├── broker-core/         # optimizer, provisioner state machine, job queue, recovery
│   │   ├── broker-cli/          # spz CLI: launch/exec/jobs/queue/status/down
│   │   ├── adapters/            # backends: aws (EC2/EKS, multi-region) first; kubernetes + slurm-ssh later phases
│   │   └── api-server/          # single-tenant REST API server (OSS/paid seam)
│   ├── platform/                # ── CLOSED SOURCE (monetized) ──
│   │   ├── accounting-service/  # usage ledger, billing, settlement
│   │   ├── dispatch-service/    # quota/fairshare/preemption policy engine
│   │   ├── observability-service/ # GPU health, metrics aggregation, alerting
│   │   └── frontend/            # web console (dashboards, team admin)
│   ├── infra/                   # CDK pipeline (Beta → BakeUsw2 canary → Prod, DrUse1 DR)
│   ├── ui-tests/ e2e-tests/ perf-tests/

License boundary: All OSS components live under packages/oss/ (Apache-2.0, exported to the public repo); all closed components live under packages/platform/ (private). The seam is the api-server interface: Platform services implement the same gRPC/REST extension points the OSS server stubs out — switching OSS → Platform is a server-URL change (SkyPilot conversion model).

Naming convention (shared AWS account)

The AWS account is a shared environment. Every CDK stack and every AWS resource MUST carry a skypurza + stage prefix. Stacks: Skypurza<Stage>... (e.g. SkypurzaBetaBroker, SkypurzaBakeUsw2, SkypurzaProdBroker, SkypurzaDrUse1). Resources: skypurza-<stage>-<component>-<purpose> (e.g. skypurza-beta-broker-state, skypurza-prod-ledger), IAM roles skypurza-<stage>-*, log groups /skypurza/<stage>/.... All names are built once in packages/oss/shared/naming.ts via a name(stage, component, purpose) helper (kwedra-roko pattern) and imported by both CDK stacks and application code — no hardcoded resource strings anywhere. Broker-provisioned runtime resources (EC2 instances, security groups, key pairs, S3 checkpoint buckets) are also prefixed skypurza-<stage>- and tagged Project=skypurza and Stage=<stage> for cost allocation and cleanup. The git repo root is also shared: every SkyPurza commit message MUST be prefixed skypurza: (enforced by convention and, once scaffolded, a commit-msg hook in the monorepo).

Language split

OSS broker packages (packages/oss/: shared schema, broker-core, CLI spz, adapters, api-server) are Python 3.11+ (pydantic models, uv-managed workspace, pytest) — chosen for SkyPilot parity and Slurm-user familiarity (D10). Closed Platform services, CDK infra, and the frontend are TypeScript (pnpm workspace). The task-YAML and API contracts are defined once as JSON Schema, generated into pydantic and zod types so the two halves cannot drift. Python unit/integration tests use pytest + moto/LocalStack; the testing strategy in section 5 applies unchanged with pytest substituting vitest for OSS packages.

3. Open-standard compliance

  • Task YAML: accept SkyPilot schema verbatim (resources.accelerators, resources.infra, num_nodes, setup, run, file_mounts, envs).
  • CLI parity: spz launch | exec | jobs launch | jobs queue | jobs cancel | status | logs | down | gpus list | check.
  • Env-var contract: export both SKYPURZA_* and SKYPILOT_* aliases (NODE_RANK, NODE_IPS, NUM_NODES, NUM_GPUS_PER_NODE, TASK_ID, JOB_RANK, NUM_JOBS) so SkyPilot-written run scripts work unchanged.
  • Slurm backend contract: zero cluster-side changes — SSH login node, generated sbatch, GRES mapping, sbatch_options passthrough, submit_as_user identity mapping.

4. Milestones

  • M0 (wk 1-2): Skeleton — monorepo scaffold, shared schema (zod) for task YAML, naming.ts, DESIGN.md v1, CI (lint/test/build), CDK pipeline skeleton.
  • M1 (wk 3-6): OSS broker MVP — AWS multi-region — AWS adapter (launch/down/status), multi-region optimizer (cheapest region with availability, capacity-error failover across regions), spot with on-demand fallback, CLI, local state store; parity tests against SkyPilot YAML corpus.
  • M2 (wk 7-10): AWS depth + managed jobs — managed jobs with auto-recovery + checkpoint buckets (S3), multi-node gang provisioning with EFA, capacity-reservation/ODCR support, autostop; Kubernetes and Slurm-SSH adapters begin only after this (moved to M4+).
  • M3 (wk 11-14): Platform seam + accounting (closed) — multi-tenant api-server extension points, usage ledger w/ settlement, showback reports.
  • M4 (wk 15-18): Dispatch + observability (closed) — Kubernetes + Slurm-SSH adapters; quota/fairshare engine, priority aging + preemption, GPU health manager, dashboards (frontend), Prometheus federation.
  • M5 (wk 19+): Hardening & GTM — SOC2-track controls, HA api-server, docs site, OSS repo public launch, Platform private beta.

Definition of done for every milestone includes its tests green in CI and its docs updated.

5. Testing strategy

  • Unit tests (per package, vitest): optimizer placement decisions, YAML schema validation, sbatch/CLI codegen, ledger settlement math (packages/platform/accounting-service).
  • Contract/parity tests: a SkyPilot YAML corpus (real examples from skypilot docs) must parse and plan identically; CLI verb parity snapshot tests; SKYPILOT_* env-var alias contract tests.
  • Adapter integration tests (hermetic): AWS via LocalStack (multi-region failover simulated by faking per-region capacity errors); kind and Slurm Docker fixtures added when those adapters land.
  • e2e-tests package: full launch→exec→recover→down flows against a dev AWS account and a kind cluster; failure-injection tests for spot preemption and node loss recovery.
  • perf-tests package: provisioning latency, 1k-job queue throughput, optimizer scaling with cluster count.
  • ui-tests package: Playwright coverage of the Platform console (dashboards, quota admin).
  • CI gates (kwedra-roko pipeline pattern): lint + unit on every PR; integration + e2e in the Beta stage; canary suite in BakeUsw2 before Prod promotion; OSS/closed import-boundary lint rule enforced in CI; a lint/CI check rejects any CDK stack or resource name not sourced from naming.ts with the skypurza- prefix.

6. Documentation

  • docs/DESIGN.md — living architecture doc (kwedra-roko convention), updated in the same PR as any design-affecting change.
  • Public docs site (OSS): getting started, task YAML reference, CLI reference, backend setup guides (AWS, Kubernetes, Slurm), SkyPilot migration/compat guide, troubleshooting.
  • Platform docs (private): accounting/chargeback model, dispatch policy language, observability dashboards, operator runbooks (HA api-server, upgrade, backup/restore).
  • Per-package READMEs + typedoc API reference generated in CI.
  • Docs-as-DoD: every milestone and every user-facing PR must update the relevant docs; CI check for stale CLI reference (generated from the CLI's own help output).

7. Risks / open questions

  • SkyPilot "open standard" is de-facto (their YAML/CLI), not a formal spec — track upstream changes; pin a compatibility version.
  • Slurm adapter correctness across Slurm versions/GRES configs (SkyPilot's own was 2 years maturing).
  • Trademark/licensing hygiene: clean-room implementation against docs/spec, no code copying despite Apache-2.0 permitting it (keeps provenance clean).
  • Closed/open package split in one workspace requires strict CI export filtering (no closed imports from OSS packages — enforce with lint rule).