# Runbook — bringing ms.gov_ked up

Everything you type to get from a fresh checkout to a working βεβαίωση. This is the single
source of truth for commands; `README.md` and `README.local.md` point here rather than
repeating them.

No PHP, Composer or node runs on the host — all of it is in containers.

---

## 0. First time only

### 0a. SSH key on Δημόκριτος' GitLab (required)

`egov/rabbitmq-classes` is fetched over **SSH (port 222)** from
`gitlab.skel.iit.demokritos.gr`, so a key registered there is mandatory before anything
installs. You need an account on that GitLab that can read `egovapp/egovrabbitmqclasses`
— ask a colleague to add you if you do not have one.

Generate a dedicated ed25519 key (the private key is what the install offers; the
`.pub` is what you register):

```bash
ssh-keygen -t ed25519 -C gov-ked -f ~/.ssh/id_ed25519_demokritos -N ""
cat ~/.ssh/id_ed25519_demokritos.pub   # copy this
```

Paste that **public** half into GitLab → Edit profile → SSH Keys → Add new key, then
verify — you must see the welcome line before continuing:

```bash
ssh -T -p 222 -i ~/.ssh/id_ed25519_demokritos -o IdentitiesOnly=yes git@gitlab.skel.iit.demokritos.gr
# expect: Welcome to GitLab, @<your-username>!
```

> **Web access ≠ git access.** Opening the repo in the browser only proves your account
> can read it. The install authenticates as a process offering an SSH key. If `ssh -T`
> answers `Permission denied (publickey)`, the key is being refused — most often an old RSA
> key that GitLab will not accept; generate the ed25519 key above and register that one.

### 0b. Dependencies

With the key in place, install Composer dependencies — all inside a throwaway container,
no host PHP/Composer needed:

```bash
./bin/install-deps.sh        # or: make install
```

The script handles the two things that bite a fresh checkout:

- **Your uid inside the container.** It builds a synthetic `/etc/passwd` + `/etc/group`
  (the image's own files plus an entry for your uid) so the process can resolve itself. This
  matters on macOS, which keeps users in Directory Service, not `/etc/passwd` — a naive mount
  dies with `No user exists for uid 501` (Linux is uid 1000). One code path covers both.
- **The SSH fetch.** It offers the key with `IdentitiesOnly` and rewrites the package URL
  from HTTPS to `ssh://git@…:222/`. The two `--ignore-platform-req` flags apply **only** to
  the composer image (newer PHP, no ext-sockets) — the app runtime is PHP 8.1 with soap and
  sockets present.

If it fails with `Permission denied (publickey)`, go back to 0a — the key is not registered.
Overrides: `KED_SSH_KEY` (default `~/.ssh/id_ed25519_demokritos`) and `KED_GITLAB` (default
`gitlab.skel.iit.demokritos.gr`).

### 0c. On Windows

This is a Laravel Sail project, and **Sail on Windows requires WSL2** — so do the whole
install from inside WSL, not PowerShell or cmd. Once is enough:

1. **WSL2** (`wsl --install` in an Administrator PowerShell, then reboot) and **Docker
   Desktop** with WSL2 integration enabled for your distro (Docker Desktop → Settings →
   Resources → WSL Integration).
2. Work **inside WSL** — launch it (`wsl`) and stay there for every step. Clone into the WSL
   filesystem (`~/demokritos-gov-ked`), **not** under `/mnt/c/…`: bind mounts are far faster
   there and the PHP toolchain behaves better.
3. Run 0a and 0b unchanged from inside WSL. `ssh-keygen`, `ssh -T` and
   `./bin/install-deps.sh` behave exactly as on Linux. Line endings are already LF everywhere
   (`.gitattributes` forces `eol=lf`), so the shell script will not break on checkout.

If your machine genuinely cannot run WSL2, stop: you would need a native PowerShell port of
the installer and could not use Sail as the rest of this runbook describes. Talk to us first.

**Environment.**

```bash
cp .env.example .env
./vendor/bin/sail artisan key:generate
```

Then hand-fill the `KED_*` credentials. They are in `docs/source_docs/credentials/` —
**two separate pairs**, one per ΚΕΔ service. Never copy the values into code, tests, fixtures
or `.env.example`.

**Pushes:** `origin` is our private mirror; `demokritos` is the client's GitLab — push
tested `main` there when there is something to deliver.

---

## 1. Bring the stack up

```bash
./vendor/bin/sail up -d      # first run builds the PHP 8.1 image (minutes)
```

That starts **four** services (`compose.yaml`):

| Service | What it is |
|---|---|
| `laravel.test` | the app — PHP 8.1, nginx, and where you run every artisan command |
| `scheduler` | `artisan schedule:work`, which fires `ked:reconcile` every `KED_RECONCILE_INTERVAL` minutes |
| `rabbitmq` | broker + management UI on <http://localhost:15672> (guest/guest) |
| `mysql` | the service is named `mysql` but runs `mariadb:10.6`, matching their production |

The scheduler starts on its own — you do not launch it, and it is easy to forget it is
running.

```bash
./vendor/bin/sail artisan migrate --seed
```

**`--seed` is not optional.** It loads the ΑΑΔΕ lookups (σχέσεις συνυποχρέωσης, ΚΑΕ, είδη
φόρου). Without them `get_lookups` returns empty dropdowns and every submission is rejected
locally as «Μη αποδεκτός ΚΑΕ».

---

## 2. The worker

The consumer is theirs, from `egov/rabbitmq-classes`. Its own terminal:

```bash
./vendor/bin/sail artisan rabbitmq:read
```

Nothing answers without it — a published command just sits on the queue.

**Killing it after a code change:**

```bash
docker compose exec laravel.test pkill -f rabbitmq:read
```

Ctrl-C on the host is not enough. Code is volume-mounted, but an already-running PHP process
keeps the old code in memory, so a host-side stop leaves a worker inside the container still
consuming with the previous version. `pkill` inside the container is the only reliable way.
RabbitMQ's management stats lag consumer changes by ~5s.

**This applies to the deployed/online worker too, not just local dev.** Observed 2026-10-08:
a lang-file fix (`lang/el/validation.php`, added to stop `Validator` from printing raw
`validation.required`-style keys) was deployed to the online environment via the normal
pipeline, but the running `rabbitmq:read` worker there had already cached "no translations for
this locale" in memory from before the deploy — same mechanism as the local case above. A full
EC2 restart fixed it (it restarts every process, worker included), but that is the heavy-handed
version of the same fix; a targeted worker restart on deploy would do it without the reboot.
**Any deploy that touches backend code, config or lang files needs the worker restarted as
part of that deploy — redeploying the code alone is not enough**, exactly as it isn't locally.

If the broker already holds a **durable** `gov_ked` queue from an older run, delete it from
the management UI — their consumer declares the queue non-durable and will not attach to a
durable one.

---

## 3. Driving it from the terminal

`bff:publish` stands in for the BFF. Second terminal:

```bash
./vendor/bin/sail artisan bff:publish ping
./vendor/bin/sail artisan bff:publish get_lookups

./vendor/bin/sail artisan bff:publish declare_signers \
  --params='{"signers":[{"afm":"003067224","firstName":"ΕΥΓΕΝ","lastName":"ΧΡΙΣΤΟΠΟΥΛΟΥ"}],"auditUserId":"u","auditUserIp":"127.0.0.1"}'

./vendor/bin/sail artisan bff:publish submit_beveosi --params=@tests/fixtures/beveosi.json
```

Once a row is `certified` (poll `get_beveosi_status` — the reply's `taftotites` carry the Τ.Ο.
per ΑΦΜ), the payments door opens:

```bash
./vendor/bin/sail artisan bff:publish get_payment_progress \
  --params='{"payId":"137569280910100000002000000001","auditUserId":"u","auditUserIp":"127.0.0.1"}'

./vendor/bin/sail artisan bff:publish get_payment_data \
  --params='{"payId":"137569280910100000002000000001","dateFrom":"2026-01-01","dateTo":"2026-12-31","auditUserId":"u","auditUserIp":"127.0.0.1"}'
```

Order matters: **nothing certifies until a signer has been declared and ΚΕΔ has accepted the
list.** `declare_signers` first, always.

There is also a `Makefile` with shortcuts, if you have `make`: `make up`, `make worker`,
`make publish CMD=ping`, `make publish CMD=submit_beveosi PARAMS=@tests/fixtures/beveosi.json`,
`make test`, `make down`, `make shell`.

The wire format, and why the `|` is always there even with no params, is in
`docs/message-contract.md`.

---

## 4. The browser

The sample React app drives the same command surface over a dev-only HTTP bridge.

```bash
# .env: DEV_BFF_ENABLED=true   (APP_ENV must be local)
# The WWWUSER/WWWGROUP prefix supplies vars Sail normally injects; npm ignores them,
# and without it compose prints a harmless "variable is not set" warning.
WWWUSER=$(id -u) WWWGROUP=$(id -g) docker compose exec -w /var/www/html/frontend laravel.test npm install
WWWUSER=$(id -u) WWWGROUP=$(id -g) docker compose exec -w /var/www/html/frontend laravel.test npm run dev
```

Then <http://localhost:5173>. The worker from step 2 must be running. If `npm install` reports
`react-router` advisories, see `frontend/README.md` — they are SSR/RSC issues that do not apply
to this client-side app.

A demo run, the failure ΑΦΜ that are worth showing, and what the app deliberately does not do:
`frontend/README.md`.

---

## 5. The scheduler

Its job is fetching the ΑΤΒ when nobody is looking at the screen — which is the normal case,
since a real ΑΤΒ only exists after the ΔΟΥ closes the day.

```bash
docker compose logs -f scheduler                 # ticks once a minute
./vendor/bin/sail artisan schedule:list          # shows ked:reconcile and when it is next due
./vendor/bin/sail artisan ked:reconcile          # run one sweep by hand
```

A healthy scheduler logs `Running scheduled tasks every minute.` and then either
`No scheduled commands are ready to run.` or a line like
`Running ['artisan' ked:reconcile] in background … DONE`. The command's own output is sent to
`/dev/null` by `schedule:work`, so run `ked:reconcile` by hand if you want to see it; failures
land in `storage/logs/laravel-<date>.log`, not on the scheduler's stdout.

Because the service overrides `command:`, it bypasses Sail's entrypoint and the process runs
as **root** rather than as `WWWUSER`. Harmless under a bind mount, where the host owns the
files anyway, but it is why a native (non-compose) deployment must run the scheduler as the
same user as the app.

Still unproven, and worth checking if you are the first to leave it running: whether
`withoutOverlapping()` holds when a sweep runs long, and whether a βεβαίωση left at
`certified_pending_closeday` overnight is picked up without anyone opening the status page.
See `docs/internal/open-points.md` §4.

**On-prem there is no compose.** The scheduler must run under supervisor (`schedule:work`) or
as a one-line cron (`* * * * * php artisan schedule:run`). Without it βεβαιώσεις never leave
`certified_pending_closeday`.

---

## 6. Fake driver or the real ΑΑΔΕ

`KED_DRIVER=fake` (the default) replays canned pilot responses and never touches the network,
so a fresh checkout demos with no credentials. `KED_DRIVER=soap` calls ΑΑΔΕ for real.

`get_lookups` reports which system is in use — `fake`, `pilot` or `production` — and the UI
shows it, because the consequences differ enormously. An endpoint that cannot be proven to be
a test system is treated as production.

Before the first real call, drive ΚΕΔ by hand:

```bash
./vendor/bin/sail artisan ked:smoke              # refuses the certifying call without --write
./vendor/bin/sail artisan ked:smoke --fresh      # six ops on the fake, from a clean slate
```

The two payment reads (`retrievePayProgress`, `retrievePaymentData`) are read-only, so they are
**not** behind `--write`. On the live pilot they are also reachable directly against a Τ.Ο. from
an earlier certified run — the 2026-09-18 evidence run left ten of them:

```bash
KED_DRIVER=soap ./vendor/bin/sail artisan ked:smoke --payId=<Τ.Ο. από το get_beveosi_status>
```

That spends exactly two ΚΕΔ calls, and it is the calibration tool for those two operations: no
Postman wire sample exists for them, so the first live answers (element names, empty-range
behaviour) get checked against the parser from here.

These ΑΦΜ exercise the failure paths under the fake driver:

| ΑΦΜ | What happens |
|---|---|
| `000000000` | ΚΕΔ rejects it. The row unlocks, stays visible in the list, and a resend reuses it and its `audit_protocol`. |
| `999999999` | No answer. The row stays **locked** at «Εκκρεμεί έλεγχος» — the debt may exist at the ΔΟΥ, so it must not be resubmitted. |
| signer `012703950` | ΚΕΔ refuses the signer list («πεθαμένος» in the spec's own pilot data). |

**Full test-data reference** — valid & invalid pilot signer ΑΦΜ and βεβαιώσεις (ΑΦΜ, ΚΑΕ,
amounts, expected outcomes), all from Παράρτημα Ε: **`docs/test-data.md`**.

---

## 6a. The audit ledger and the ΑΑΔΕ evidence pack

Every outbound ΚΕΔ call opens a `ked_audit_records` row before the wire hop and completes it
afterwards (`SUCCESS` / `ERROR`; calls whose worker died mid-flight are marked `LOST` by the
sweeper inside `ked:reconcile`). `ked_call_log` still keeps the full masked envelopes — the two
tables are the journal and its index. The audit ids come from `ked_audit_sequences`:
`auditTransactionId` counts per method («1», «2», …), `auditProtocol` is `{n}/DD-MM-YYYY` minted
once per βεβαίωση and reused on resends (Παράρτημα Ζ).

The audit identifiers travel with **no spaces**: `KED_AUDIT_UNIT=EKEFEDEMOKRITOS` in `.env`
(the config default matches — an old `.env` carrying `EKEFE DEMOKRITOS` overrides it, so edit it
by hand).

When ΑΑΔΕ must be shown the pilot testing — the step before production credentials — print the
ledger and send it:

```bash
./vendor/bin/sail artisan ked:audit-report                     # markdown, straight into the email
./vendor/bin/sail artisan ked:audit-report --from=2026-09-18 \
    --out=storage/app/ked-pilot-evidence.csv                   # CSV in the template ΑΑΔΕ accepted
```

The CSV follows the template ΑΑΔΕ accepted from katoikia
(`docs/source_docs/audit/audit_records_202609031539.csv`): same column order minus their async
job columns, `user_id` carrying the audit user, and `audit_transaction_date` rendered in local
time with offset. The markdown additionally pairs every row with ΚΕΔ's `callSequenceId` and the
βεβαίωση's ΑΦΜ / ΚΑΕ / mcNo / ΑΤΒ, plus a per-method summary. Credentials are authorised per
operation, so the evidence wants at least one successful call per method — one `declare_signers`
plus full submit → reconcile lifecycles cover the certification four, and the two payment reads
(`retrievePayProgress`, `retrievePaymentData`) appear in the per-method summary once
`ked:smoke --payId` has run against the pilot — and one deliberate error case to show
rejections are audited too.

---

## 7. Tests, and shutting down

```bash
./vendor/bin/sail test
./vendor/bin/sail down
```

The suite runs entirely against the fake driver and touches no network.
