# Web server (remote use) The web site lets the students work on the labs from a browser, without the Qt interface and without a remote desktop: the projects run on a **pool of Docker hosts**, every tab of the graphical interface (Schema, Informations, Machines, Networks, Questions, Evaluation, Terminals, Apply Configuration, Log) is rendered in the browser and the terminals go through WebSockets. The users log in with an LDAP directory (the portal binds to it, or an Apache reverse proxy in front authenticates them), or everybody is anonymous. Nothing of a lab ever reaches the browser but what a student sees today: the content of `info.json` (resolved to the user's language, the instructor texts stripped), the outputs of the `sre --user` commands (the schema, the grades of a self-evaluation, the console of a machine). The lab files, `data.json`, the hidden machines and the archives stay on the pool hosts. The instructors get, on top of what a student sees (*Instructors* below): - a **Watch page**, the web counterpart of `sre watch`: every running project of the pool, one row per project with its owner, its lab, the last time the student did something on it, the grade and mark of its last evaluation, its age and its errors, the number of self-evaluations the student has run, refreshed every few seconds; - an **Archives page**: every evaluation of the pool hosts, the closed projects included, kept in a database (the results, never the answers), searched by student, group, lab, host and period, with the detail of an evaluation and the summary matrix of a lab; - the **projects of the students**, open to them to help: every tab of the project, and the student's terminals joined when the Terminals tab opens, so that the instructor and the student see the same shell and both type in it (a private terminal, unseen by the student, is offered too); - **presentations**: a project, the instructor's own or a student's, duplicated read-only on the browsers of the whole class or of chosen students and groups to demonstrate something. What the instructor does on their own page (changing tabs, typing in a terminal, selecting a question, changing the drawing of the schema, …) is shown at once on the students' browsers; - the **focus** on one student: while helping a student on their project, the instructor brings the student's browser to that project and keeps the two pages on the same tab, the same terminal and the same question, whichever of the two changes it; - the **web conference**: an instructor opens a BigBlueButton room for chosen students or groups (or the whole class) and talks to them, camera and screen if they wish, in a window over the pages of the site while everybody goes on working (*Web conference* below). ## Architecture ``` browser ──HTTPS──► sre-web (the portal) ──HTTP + token──► sre-agent (one per pool host) ──► sre --user … ``` - **`sre-web`**, the portal, is the web site the browsers talk to. It authenticates the users, keeps a signed session cookie, knows the pool of hosts and relays every request about a project to the agent of the host running it. It runs as an unprivileged user, needs neither Docker, Kathara nor the labs, and may run anywhere on the network (the Debian package `sysreseval-web` installs alone). - **`sre-agent`** runs on every pool host, as root, next to `sre`. It drives the `sre` CLI in user mode on behalf of the web users exactly as `sre-wrapper` + `sudo` do for the students of a workstation (`sre --user ` with `SUDO_USER=`), owns the terminals (a pseudo-terminal per `sre connect`), writes the answers file with the user's identity (atomically and under the advisory lock on the `answers/` directory the Qt GUI takes too, the owner of a file the Qt GUI of the same student created kept, so that the GUI goes on writing it; what that GUI writes is pushed to the open pages, which show the same answers), merges the cheat answers of the states, tails the operations log and schedules what the Qt GUI used to schedule: the periodic evaluations of a lab (`eval_interval_without_exam_mode`), the evaluation before a project is closed (`eval_before_exit`), and an idle reaper closing the projects nobody touched for the configured time. For an exam project of the web site (*Exams* below) it applies the rules of the Qt GUI's exam mode to that project alone: the evaluations at the period of the exam, the final one at its end, the owner cut off then. - **`sre-web-tool`** is the command line next to the portal: `sre-web-tool …` lists and changes the pool of the running portal through its administration API (*The pool* below). - The pool hosts share the lab tree as today (NFS, rsync or the Debian package) and, if the instructor wants `sre watch` / `sre sheet` / `sre outline` across the pool, an `archive_dirs` entry on a shared directory. The `hostname` of the archives is the pool host. The Watch page of the instructors needs none of that: each agent reads the archives of its own host (`params.archive_dirs`) on demand. - Placement: a new project goes to the host where its user has a local login session (the workstation they sit at, `follow_local_sessions`), else to the reachable host with the lowest `projects / max_projects` (then the lowest load); a host in exam mode is never used, a host with `instructors_only = true` never for a student (an instructor may also name the host on the Open page); the whole rule is under *Placement* in *A pool of workstations*. A classroom workstation holds about thirty projects (210 containers in 3.4 GB). - The lab list of the Open page comes from one host, the one a new project of the user would go to (every host of a pool serves the same lab tree; the next candidate is asked when it fails), and is kept `labs_cache_ttl` seconds (`60`) per role, since the reserved hosts are the instructors' only: opening the page costs one `sre list` per minute whatever the size of the pool. The agent enforces what the `sre` CLI does not check: the exact running lab name (the CLI resolves substrings) and its owner (a user only reaches their own projects, an instructor every project; the CLI's own check, `params.check_project_owner`, lets a `--user` caller reach their own projects and root's only, which is why an instructor is served by privileged calls; the `wipe`, `stop-all`, `exec` and host-wide exam commands have no endpoint: an exam of the web site is a `POST /v1/projects` with an `exam` object, adjusted with `PUT` / `DELETE …/exam`), `allow_connection` of the devices, `allow_self_grade`, the user-allowed states, the `export_schema_pdf` flag of the lab (the *Export Schema to PDF* button). Save, restore and the Kathara export are not offered on the web. An instructor acting on somebody else's project is served by a **privileged** `sre` (no `--user`, their own login: `sre eval` without cooldown nor counter, `sre connect` to every listed device, `sre state`), never by a command run as the student; the owner, instructor or not, follows the student path. ## Installation ### From a checkout `make venv` installs the Python dependencies (`aiohttp`, `jinja2`, `aiohttp-jinja2`, `ldap3`, and `pymysql` for the optional database of the connection log); `make xterm` and `make novnc` (run by `make install`) download the terminal library and the VNC viewer of the browser into `web/static/vendor/` (checksum-pinned, not committed). `scripts/install.sh` offers to install the systemd units `sre-agent.service` and `sre-web.service` (disabled) and the example configuration files `agent.toml` and `web.toml` in `/etc/sreweb/` (`params.agent_config_file` / `params.web_config_file`; `--config` names another file). On every pool host: write `/etc/sreweb/agent.toml` (root, mode 0600) with a long random token (the example file listens on every interface, `0.0.0.0:8451`: `allowed_clients` and the mutual TLS below keep the port to the portal), then `systemctl enable --now sre-agent`. The unit starts the agent only when that file exists and `/etc/sreweb/DO_NOT_START_AGENT` does not (`ConditionPathExists=`: one image for a whole classroom may ship the unit enabled, the hosts without a configuration file, or with the marker, skip it without any failure; checked at boot and on `systemctl start` / `restart`), and it does not restart the agent after a configuration error (exit status 78, `EX_CONFIG`: a bad `agent.toml`, a missing certificate; the message is in the journal, fix the file then `systemctl restart sre-agent`); five failed starts within five minutes leave it stopped until the next boot or `systemctl reset-failed`. For the Display tab (below), install `tigervnc-standalone-server` and `openbox`, create the `sreweb-x11` system user (`scripts/install.sh` offers it) and set `[display] enabled = true`. For a whole classroom, `scripts/make-web-hosts.sh` (see *A whole site at once* below) writes the tokens, the certificates and the `[[hosts]]` entries from a list of machines. On the portal host: create the `sre-web` system user (`adduser --system --group --no-create-home --shell /usr/sbin/nologin sre-web`), write `/etc/sreweb/web.toml` (root:sre-web, mode 0640: it holds the cookie secret and the agent tokens; the files it imports likewise), then `systemctl enable --now sre-web`. Put a reverse proxy doing TLS in front (`scripts/etc/sre-web-nginx.conf` is an nginx example, `sre-web-apache.conf` an Apache one that also authenticates the users: the proxy mode below), or give the portal its own certificate (`tls_cert` / `tls_key`). ### Debian packages `sysreseval-agent` on every pool host (it depends on `sysreseval` and `sysreseval-web`, whose code it shares), `sysreseval-web` on the portal host (it depends on nothing of SysResEval). The units are `sysreseval-agent.service` and `sysreseval-web.service`, disabled by default (the agent's with the conditions and restart rules above, on `/etc/sysreseval/agent.toml` and `/etc/sysreseval/DO_NOT_START_AGENT`); the configuration files are `/etc/sysreseval/agent.toml` and `/etc/sysreseval/web.toml` (examples in `/usr/share/doc/sysreseval-*/examples/`); the postinst of `sysreseval-web` creates the `sre-web` user and protects `web.toml` and `web.d/*.toml` when they exist. The postinst of `sysreseval-agent` creates the `sreweb-x11` user of the virtual X desktops; `tigervnc-standalone-server` and `openbox` are its Recommends (the Display tab is off until `[display] enabled = true`). ## Configuration ### `agent.toml` | Key | Default | Meaning | |---|---|---| | `listen` | `127.0.0.1:8451` | the address the portal reaches: `0.0.0.0:8451` in the example file (every interface, the Docker bridge the students' containers reach included: set `allowed_clients` and `tls_ca`), or a management interface the students do not reach | | `token` | required | the shared secret of this host, repeated in the portal's `[[hosts]]` entry (32+ characters) | | `sre_exe` | the installation's `sre` | the command run for the users | | `poll_interval` | `1` | seconds between two scans of the projects directory (the Qt GUI polls every second too) | | `max_idle` | `24h` | default idle timeout of a project (the portal gives its own per user); `0` = never | | `terminals_per_project`, `terminals_per_device` | `16`, `4` | caps on the open terminals (every Connect opens a new terminal on the machine); at a cap, the idle terminal that has waited longest gives way | | `terminals_linger` | `120m` | how long a terminal whose last browser left (the page closed or reloaded, another project opened, the network cut) is kept for the browser to come back to it; `0` = it ends at once; the × of a terminal ends it at once in any case | | `max_concurrent_starts`, `max_concurrent_evals` | `2`, `2` | concurrent `sre start` / `sre eval` on the host | | `lab_list_cache_ttl` | `60` | seconds the output of `sre list` is cached | | `state_dir` | `/var/lib/sre/web-agent` | the idle clocks of the projects, kept across restarts | | `tls_cert`, `tls_key` | none | the certificate and key of this agent (TLS to the portal) | | `tls_ca` | none | the CA a **client certificate** must be signed by: only the portal gets past the handshake (mutual TLS) | | `allowed_clients` | `[]` (any) | the addresses or networks the agent answers, e.g. `["10.0.0.1", "10.0.1.0/24"]` | | `occupied_check` | `both` | when the host reports itself *occupied* (no new project, never powered off): `sessions` (a local login session: `loginctl`, or `who`), `projects` (projects of the Qt GUI on the host), `both`, `never` | | `close_projects_on_stop` | `shutdown` | what the agent does with the projects it manages when its unit stops: `shutdown` closes them (last evaluation, `sre stop`) only when the system is going down, `always`, `never`; within `shutdown_timeout` (`240` s, the unit's `TimeoutStopSec` is 300) | | `allow_power_off` | `false` | the portal may power the host off (`POST /v1/power/off`, refused when occupied or with any project) with `power_off_command` (`systemctl poweroff`) | | `restart_command` | none | how the agent restarts itself on `sre-web-tool restart` (`POST /v1/restart`): by default `systemctl restart` of its own unit, read from `/proc/self/cgroup`; a command for an agent run otherwise (refused, 409, when neither applies) | | `[timeouts]` | `start = 900`, `eval = 600`, `state = 600`, `stop = 300`, `list = 30`, `report = 600` | seconds, per `sre` command (`report`: the spreadsheet / PDF reports of the Archives page, built by `sre report` on this host; `export`, the timeout of the former Kathara export, is still accepted) | | `[display]` | `enabled = false` | one virtual X desktop per project whose lab has an `x11_host` machine, shown in the Display tab (see *The Display tab* below): `user` (`sreweb-x11`, the unprivileged account the X processes run as), `xvnc_command` (`Xvnc`, TigerVNC), `window_manager_command` (`openbox`, `""` = none), `geometry` (`1280x800`), `depth` (`24`), `frame_rate` (`30`), `first_display` / `last_display` (`10` / `99`: display `:N` listens on TCP port 6000+N for the containers), `runtime_dir` (`/run/sre-agent`: the RFB sockets and X authority files), `start_timeout` (`10` s for the X server to come up) | | `[preload]` | `at_start = false` | the Docker images of the labs pulled by the agent (`sre preload-images`) when the portal gives the host its turn (see *Pre-loading the images* below): `at_start` (the agent asks for a preload each time it starts; `sre-web-tool preload` asks by hand in any case), `paths` (the lab files and directories scanned; default: the lab directories of the installation), `timeout` (`"2h"`: a preload still running after this long is stopped) | Durations are written as `90`, `"90s"`, `"30m"`, `"2h"` or `"1d"`. An unknown key is refused. `import` (a path or a list of paths relative to `agent.toml`: a directory, a glob or a file) merges other files into this one, as for `web.toml` below. ### `web.toml` | Key | Default | Meaning | |---|---|---| | `listen` | `127.0.0.1:8450` | behind a reverse proxy: the loopback | | `public_url` | none | the URL the users type (the `Origin` of the WebSockets is checked against it) | | `secret` | required | signs the session cookies (32+ random characters) | | `trusted_proxies` | `[]` | proxy mode: the addresses or networks of the reverse proxy, the only peers whose identity headers are trusted (the TCP peer address, never `X-Forwarded-For`); required in that mode. In every mode, the connection log takes the client address from the `X-Forwarded-For` header of these peers | | `languages` | `["en", "fr"]` | the languages offered at login, the first one by default | | `tls_cert`, `tls_key` | none | own TLS instead of a proxy | | `[auth] mode` | `anonymous` | `anonymous`: a pseudonym `guest-xxxxxxxx` per browser (the user may type a display name); `ldap`: the directory account; `proxy`: the identity the reverse proxy passes in headers (below) | | `[auth] administrators` | `[]` | the logins of the administrators: instructors who may read the connection log on the History page (*Administrators* below); a login that is not an instructor gets nothing. By group: `administrator_groups` / `administrator_group_filter` of `[auth.ldap]`, `administrator_groups` of `[auth.proxy]` | | `[auth.ldap]` | | `uri`, then either `bind_dn_template = "uid={login},ou=people,…"` (bind as the user) or `search_base` + `search_filter` (+ `bind_dn` / `bind_password` of a service account) to find the user first; `attr_fullname` (`cn`), `attr_mail` (`mail`), `attr_group` (none; the attribute holding the student group, e.g. `departmentNumber`, its first value: the Group column and filter of the Watch page); the instructor tests (below): `instructor_groups` (DNs; `memberOf`, or `group_filter = "(member={dn})"`), `instructor_filter` (`"(gidNumber=2000)"`, `""` = none), `group_search_base` (none = no membership test) + `instructor_group_filter` (`"(&(objectClass=posixGroup)(gidNumber=2000)(memberUid={login}))"`); the administrator tests (*Administrators* below): `administrator_groups` (DNs, tested like `instructor_groups`) and `administrator_group_filter` (a group search under `group_search_base`, none by default); the connection: `ldaps://` or `ldap://` + `start_tls`, the server certificate always verified against the system CA store or `tls_ca` (the absolute path of a PEM file: a CA bundle, or the self-signed certificate of the server itself) and the host of the `uri` against the names of the certificate (`subjectAltName`, else the CN: name the server as its certificate does, not by address); a plain `ldap://` without `start_tls` is refused at startup unless `allow_cleartext = true` (the passwords in LDAP mode, the roles, groups and administrators in every mode would travel unprotected); `timeout`. The login attribute: `attr_login` (none = the attribute of a `search_filter` of the form `(attribute={login})`, else the RDN attribute of `bind_dn_template`); in LDAP mode the identity carries its stored value, whatever case the user typed (the directory matches `Bob` for `bob`: one user, not two), and the Groups page (*User groups* below) lists the people by it, in every mode, with `groups_filter` (`"(objectClass=posixGroup)"`: the groups listed under `group_search_base`; `""` = none), `attr_group_name` (`cn`), `attr_group_members` (`memberUid`; a value may be the login, the `uid` or the DN of a person). Also read in proxy mode with `ldap_roles` / `ldap_groups` | | `[auth.proxy]` | | `header_login` (`X-Remote-User`), `header_fullname` (`X-Remote-Name`), `header_email` (`X-Remote-Email`), `header_groups` (`X-Remote-Groups`): the headers set by the proxy; `groups_separator` (`;`); `header_group` (none; the header holding the student group, e.g. `X-Remote-Group` from `departmentNumber`, its first value); `instructor_groups`: the values of the groups header giving the instructor role; `instructor_header` + `instructor_values` (e.g. `X-Remote-Gid` and `["2000"]`): a second header test; `administrator_groups`: the values of the groups header making an instructor an administrator; `ldap_roles` (`false`): the portal runs the tests of `[auth.ldap]` itself for the login the proxy gives, and reads its `attr_group`, the result cached per login for `ldap_roles_ttl` (`10m`); `ldap_groups` (`false`): the group part alone (`attr_group` of `[auth.ldap]` for a user whose group header is empty, and for the owners of the Watch page the agents know no group for; the roles stay those of the headers; `ldap_roles` implies it); `logout_url`: where Logout sends the browser (the `/apache-logout` location of the Apache example; none: no Logout button) | | `[pool]` | | `idle_timeout_anonymous` (`2h`), `idle_timeout_authenticated` (`24h`): the idle timeout given to the agents per user, `0` = never; `poll_interval` (`2`): seconds between two polls of the agents; `ca`: the CA the agents' certificates are verified against (the system store without it); `client_cert`, `client_key`: the client certificate of the portal, presented to every agent; the elastic pool (below): `spare_projects` (`5`), `min_hosts_up` (`1`), `active_hours` (`07:00-20:00`, local time, `""` = always), `wake_timeout` (`180`), `wake_interval` (`30`), `wol_broadcast` (`255.255.255.255`), `idle_host_timeout` (`0` = never); `follow_local_sessions` (`true`): a user logged in at the keyboard of a pool host gets their new projects there whatever its load, below its `max_projects` (*Occupied hosts* below); `labs_cache_ttl` (`60`): seconds the lab list of the Open page, asked from one host, is kept per role (`0` = asked at every open); `max_preloads` (`2`): the hosts pulling the Docker images of the labs at the same time (*Pre-loading the images* below; `0` = the portal lets none) | | `[limits]` | | `eval_interval` (`10` s between two self-evaluations of a user), `start_interval` (`30` s between two starts of a user; a start that created no project — the flavor form shown or refused, no host available, the agent not answering — gives the slot back at once), `login_per_minute` (`5` per client address), `max_answer_bytes`; in anonymous mode `eval_interval` and `start_interval` count per client address too, since a new pseudonym costs one `POST /login` | | `[admin]` | | `token`: the token of `sre-web-tool …` (32+ characters; none = the command is refused), accepted from the loopback only unless `listen_any = true`; `state_file`: what is changed by hand in the pool (the hosts added, removed, enabled, disabled, drained), kept across the restarts of the portal (`/var/lib/sre-web/pool.json` when that directory exists, else `pool.json` next to `web.toml`; *Restarting the portal* below) | | `[access_log]` | | the connection log (below): `file` (none; the absolute path of the log file, e.g. `/var/log/sre-web/access.log`, rotated daily), `keep` (`400` days of rotated files kept; `0` = no rotation, for logrotate), `journal` (`true`: the lines also go to the portal's own log: stderr, the journal, `docker logs`), `session_timeout` (`30m` of inactivity closes a session); `database` (the next row) | | `[access_log.database]` | none | the same events in a MariaDB / MySQL table (below): `host` or `unix_socket`, `port` (`3306`), `user`, `password` (in an imported secrets file), `name` (the database), `table` (`access_log`, created when missing), `keep` (`400` days of rows kept, purged daily; `0` = forever), `queue` (`10000` rows kept in memory while the server does not answer), `connect_timeout` (`5`); needs PyMySQL (`python3-pymysql`, a *Suggests* of the Debian package: optional) | | `[archives]` | | the archives of the evaluations of the pool hosts in a second table of that database (*The archives of the evaluations* below; on iff `[access_log.database]` is set): `table` (`archives`, created when missing), `keep` (`0` = the rows are kept forever; else days of rows kept, purged daily), `poll_interval` (`30s` between two collections from the agents) | | `[groups]` | | the user groups of the Groups page (*User groups* below): `table` (`user_groups`: the custom groups the instructors create, in the database of `[access_log.database]`, their members in `_members`; both created when missing), `directory_ttl` (`10m`: how long the people and groups read from the directory are kept), `refresh_interval` (`10s`: the least time between two reads forced by the *Refresh* button of the page; the clicks arriving during a read share it, `0` = none) | | `[exams]` | | the exams of the Exams page (*Exams* below; on iff `[access_log.database]` is set): the defaults of a new exam, each adjustable per exam — `lead` (`10m`: how long before the start the other projects of the students are hidden and the countdown shown), `pre_start` (`5m`: how long before the start their exam projects are started in the background), `eval_interval` (`60s`: the period of the evaluations the agents run during the exam) —, `close_after` (`1h`: how long after the end of a student's exam their exam projects are closed; `0` = never, an instructor closes them), `table` (`exams`, with `
_members`, both created when missing), `keep` (`0` = the past exams are kept forever; else days, purged daily) | | `[site]` | none | the institution shown next to the SRE logo in the header of every page, the login page included: `logo` (the absolute path of an image file, JPG / PNG / SVG…, readable by `sre-web`, served at `/site/logo`), `name` (a string, or one per language: `{ fr = "IUT de Démo", en = "Demo Institute" }`, the first configured language when the page's one is missing), `url` (the page the two link to; none: plain text) | | `[bbb]` | none | the BigBlueButton server behind the **Webconf** button of the instructors (*Web conference* below): `url` (the server, or its `bigbluebutton/` / `bigbluebutton/api/` path) and `secret` (the shared secret of its API, `bbb-conf --secret`; both or none), `checksum` (`sha1`; `sha256` or `sha512` when the server is restricted to them), `welcome` (the welcome message of the client: a string, or one per language; the server's message by default), `record` (`false`), `mute_on_start` (`false`), `duration` (`0` = unlimited; a duration, `"2h"`: the server ends the meeting after it), `max_participants` (`0` = unlimited), `poll_interval` (`10s`: how often the participants and the ended meetings are read from the server), `[bbb.userdata]` (the `bbb_*` client settings sent with every join, e.g. `bbb_skip_check_audio = true`), `private_rooms` (`true`: the **Talk privately** button of an instructor on a student's project, *Private conversations* below; `false` removes it), `[bbb.private_userdata]` (the `bbb_*` client settings of the joins of a private conversation, over `[bbb.userdata]` and the portal's own) | | `[[hosts]]` | required | one per pool host: `name` (what the URLs show), `url` of its agent, `token` of its `agent.toml`, `max_projects` (`25`); to wake it, `mac` (Wake-on-LAN) or `wake_command` (`{name}`, `{mac}` substituted); `power = true` lets the portal power it off; `enabled = false` keeps it polled without placing projects; `instructors_only = true` reserves it for the instructors' projects (a student is never placed there and does not see its labs; a project already running there stays reachable; outside the elastic pool, see *Reserved hosts*) | | `import` | none | other files merged into this one before it is checked: a path or a list of paths, relative to the directory of `web.toml`: a directory (its `*.toml` files, by name), a glob pattern (`"web.d/*.toml"`, possibly matching nothing) or a file. Tables merge, `[[hosts]]` entries add up; a key given in two files is refused, naming both; an imported file cannot `import` | `import = ["web.d"]` splits the configuration into `/etc/sysreseval/web.toml` and the `*.toml` files of `/etc/sysreseval/web.d/`, read by name after it: a file per pool host, the secrets apart. Nothing is silently overridden: a key given twice is an error naming both files, and an error found after the merge names `web.toml`. The imported files hold the same secrets as `web.toml` and need the same owner and mode (root:sre-web, 0640). The full name and the e-mail of an LDAP account (`attr_fullname`, `attr_mail`) go into the answers file of the projects like the GECOS fields do on a workstation, so `sre outline` and `sre sheet` show them; an anonymous user is identified by the pseudonym and the display name. The student group (`attr_group` in LDAP mode, `header_group` or `ldap_roles` / `ldap_groups` + `attr_group` in proxy mode; `departmentNumber` typically, its first value, capped at 64 characters) goes there too under the key `group` (hence into the archives): the Watch page shows it and filters on it. With a directory the portal can read (`attr_group` in LDAP mode, `ldap_roles` or `ldap_groups` in proxy mode), the Watch page also looks up the group of the owners the agents know none for, first of all the projects started on the Qt GUI, whose answers file never carries one, and shows for the projects of the Qt GUI the full name of the directory (`attr_fullname`) rather than the GECOS name of the account their answers file carries (lower case on some sites): one cached lookup per login. The agent writes that metadata as soon as a project is created, before any answer, and again when the owner opens the project page and the file lacks it or holds other values (so a project created by an older agent, or on the Qt GUI, gets its name and group then); a request whose identity carries no full name or no group (a header the proxy did not set) never erases the stored values. ### Authentication by the reverse proxy With `[auth] mode = "proxy"` the portal has no login page: the reverse proxy in front authenticates the users itself and names them in request headers, which the portal reads on every request (pages, API calls, the NDJSON streams of the starts, the WebSockets of the terminals). Nothing is stored in a cookie: the proxy's session is the session, and a request that does not come from a peer of `trusted_proxies` has no identity whatever its headers. `scripts/etc/sre-web-apache.conf` is the Apache 2.4 example (`mod_authnz_ldap` + `mod_proxy`): ```apache AuthType Basic AuthName "SRE" AuthBasicProvider ldap AuthLDAPURL "ldaps://ldap.example.org/ou=people,dc=example,dc=org?uid,cn,mail,memberOf,gidNumber,departmentNumber?sub?(objectClass=inetOrgPerson)" AuthLDAPRemoteUserAttribute uid Require valid-user RequestHeader set X-Remote-User "expr=%{REMOTE_USER}" RequestHeader unset X-Remote-Name RequestHeader set X-Remote-Name "%{AUTHENTICATE_CN}e" env=AUTHENTICATE_CN RequestHeader unset X-Remote-Email RequestHeader set X-Remote-Email "%{AUTHENTICATE_MAIL}e" env=AUTHENTICATE_MAIL RequestHeader unset X-Remote-Groups RequestHeader set X-Remote-Groups "%{AUTHENTICATE_MEMBEROF}e" env=AUTHENTICATE_MEMBEROF RequestHeader unset X-Remote-Gid RequestHeader set X-Remote-Gid "%{AUTHENTICATE_GIDNUMBER}e" env=AUTHENTICATE_GIDNUMBER RequestHeader unset X-Remote-Group RequestHeader set X-Remote-Group "%{AUTHENTICATE_DEPARTMENTNUMBER}e" env=AUTHENTICATE_DEPARTMENTNUMBER RequestHeader unset Authorization # Logout: a location that knows no user, so the browser forgets the credentials it cached ProxyPass "/apache-logout" ! AuthBasicProvider file AuthUserFile /dev/null Require valid-user ErrorDocument 401 "... You are logged out. Log in again ..." ProxyPreserveHost On ProxyPass "/" "http://127.0.0.1:8450/" upgrade=websocket timeout=3600 ProxyPassReverse "/" "http://127.0.0.1:8450/" ``` `AuthLDAPRemoteUserAttribute` is not optional: it makes `REMOTE_USER`, hence the login header, the value the directory stores in the login attribute. Without it Apache passes what the user typed, and since the directory matches `Bob` for `bob` (`uid` is caseIgnoreMatch) one account would become two users of the portal (separate projects, out of the exam audiences and custom groups naming `bob`, split Watch and Archives rows). The attribute must be the one the agents and the exam audiences know the users by (`uid` for the Unix login of the workstations, see *A pool of workstations*). In LDAP mode the portal does this itself: the identity carries the stored value of the login attribute (`attr_login`, else the attribute of the `search_filter` or of the `bind_dn_template`), whatever case the user typed. The rules that keep this safe: - The portal listens on the loopback (or on an address only the proxy reaches) and `trusted_proxies` holds the proxy's address as the portal sees it (`127.0.0.1` and `::1` for a proxy on the same host); the portal takes the TCP peer address, never `X-Forwarded-For` (the connection log alone reads that header, from these peers only, for the client address). - The proxy sets the four headers on **every** request, replacing whatever the browser sent (`RequestHeader set` does; never pass a client header through), after the authentication (not `early`). An optional header is removed when the attribute is missing, or `mod_headers` writes the literal `(null)`, which the portal treats as empty. - The attributes exported by `mod_authnz_ldap` (`AUTHENTICATE_CN`...) are those listed in the `AuthLDAPURL`; a multi-valued one is joined with `; ` (hence `groups_separator = ";"`). The groups header may carry any attribute, `memberOf` DNs or `eduPersonAffiliation` values: `instructor_groups` lists the values giving the instructor role; `instructor_header` / `instructor_values` test a second attribute the same way (`gidNumber` above, the primary group); `header_group` names the header carrying the student group (`departmentNumber` above). A group membership the proxy cannot export (a `posixGroup` listing its `memberUid`, without the memberof overlay) is tested by the portal itself with `ldap_roles = true` and the `[auth.ldap]` settings (see *Who is an instructor*). A long list of groups can exceed Apache's `LimitRequestFieldSize` (8190 bytes): raise it or export fewer attributes. - `public_url` must be the `https://` URL the users type (the `Secure` flag of the language cookie, the `Origin` check of the WebSockets together with `ProxyPreserveHost On`). - An HTTP Basic authentication has no session to end, but the browser forgets the credentials it cached as soon as the server refuses them: the `/apache-logout` location of the example knows no user (`mod_authn_file` on `/dev/null`, the realm inherited from the main location), so every request gets a 401 and the browser shows its password dialog once more; cancelling it displays the "logged out" page of the `ErrorDocument` (a password typed there is refused and the dialog comes back). `logout_url = "/apache-logout"` makes the Logout button appear; the button is hidden without it. A `mod_auth_form` login (sketched at the end of the example) has a real logout handler at the same URL. - `sre-web-tool …` talks to the loopback port directly, never through the proxy (the Basic authentication would refuse its bearer token). ### The connection log `[access_log]` of `web.toml` makes the portal keep a log of its use: who connected from where and for how long, which projects were opened, started, evaluated, closed. One `key=value` line per event (a value with spaces or quotes is JSON-quoted, so a user agent or a display name cannot forge a line), in the file named by `file` (rotated at midnight, `keep` days of `access.log.YYYY-MM-DD` files kept; `keep = 0` leaves the rotation to logrotate, the file is reopened when it moves) and, with `journal = true` (the default, and the whole log when no file is set), in the portal's own log on stderr: the journal under systemd, `docker logs` in a container. The unit creates `/var/log/sre-web` (`LogsDirectory=`); a container mounts a directory for the file or writes it under `/var/lib/sre-web`. The lines of the file start with a timestamp (`2026-10-26T09:12:03+0100`): ``` event=login login=alice addr=10.0.1.7 mode=ldap role=student anon=0 name="Alice Test" group=INFO1 event=session_open login=alice addr=10.0.1.7 role=student anon=0 name="Alice Test" group=INFO1 ua="Mozilla/5.0 …" event=start login=alice addr=10.0.1.7 lab=sre/dhcp.py instructor=0 debug=0 host=a01 placement=load result=created rln=1761466323@@@dhcp@@@alice status=200 duration=12.4 event=project_open login=alice addr=10.0.1.7 host=a01 rln=1761466323@@@dhcp@@@alice event=terminal_open login=alice addr=10.0.1.7 host=a01 rln=1761466323@@@dhcp@@@alice device=m1 join=- private=0 event=terminal_close login=alice addr=10.0.1.7 host=a01 rln=1761466323@@@dhcp@@@alice device=m1 join=- private=0 duration=612.3 code=1000 event=eval login=alice addr=10.0.1.7 host=a01 rln=1761466323@@@dhcp@@@alice status=200 duration=8.1 event=project_close login=alice addr=10.0.1.7 host=a01 rln=1761466323@@@dhcp@@@alice duration=1803.6 code=1001 event=stop login=alice addr=10.0.1.7 host=a01 rln=1761466323@@@dhcp@@@alice status=200 duration=3.0 event=logout login=alice addr=10.0.1.7 status=200 duration=0.0 event=session_close login=alice addr=10.0.1.7 role=student name="Alice Test" group=INFO1 reason=logout since=2026-10-26T09:11:50+0100 duration=1850 requests=14 projects=1761466323@@@dhcp@@@alice ``` A **session** is the requests of one login from one client address (`addr`): it opens at the first request (`session_open`: role, pseudonym or not, name, student group, user agent) and closes (`session_close`) at a logout, after `session_timeout` without a request (`reason=idle`; the duration then ends at the last request, not at the expiry) or when the portal stops (`shutdown`); a session whose project page, terminal or desktop is still open never expires. The closing line repeats the role, the name and the group (the student group of the directory or of the proxy, `attr_group` / `header_group`, `-` without one; the last non-empty value seen during the session, since a directory lookup may fail on the first request) and gives the start, the duration in seconds, the number of requests (the polls of the pages and the keep-alive of a project are not counted) and the projects used: the *Sessions* view of the History page (*Administrators* below) is built from that line alone. In proxy mode there is no login: the first request of a user opens the session and the request following a logout opens another one. The other events: - `login` (`mode`: `anonymous` or `ldap`) and `login_failed` (`reason`: `password`, `rate_limited`, `directory` when the directory does not answer, `proxy_mode`); - `start`: the lab, the host chosen and `placement`, why that host (`load`: the least loaded, `session`: the user is logged in at its keyboard, `named`: the instructor named it), whether the instructor mode or the debug mode was asked (`instructor`, `debug`), `result` (`created`, `flavor_form`, `flavor_error`, `error`) and `rln`, the running lab name of the new project; - `project_open` / `project_close`: the project page, open as long as its events WebSocket is (`duration` in seconds, `code`: the WebSocket close code; `view=instructor` for the instructor view); `terminal_open` / `terminal_close` (`device`, `join`: the session joined, `private`) and `display_open` / `display_close` likewise; a socket refused before it opens (401, 403, 404) gives a close line with `status` instead of `code`; - `eval`, `state` (`state`), `stop`, `display_start`: the HTTP status and the duration; - `denied`: a call answered 401, 403 or 429 (`method`, `path`), `login=-` when nobody was logged in; `admin`: a call of `sre-web-tool` changing the pool, or an action of the Pool page of an instructor (their `login`); `history`: an administrator read the log on the History page (*Administrators* below; the filters asked come as `login_filter`, `addr_filter`, `event_filter`, `host_filter`, `rln_filter`, `lab_filter`, `group_filter` (the key of the group, `kind:name`), `since_filter`, `until_filter`, `before_filter`; `view=sessions` for the Sessions view, `view=statistics` for the Statistics view with the period resolved in `since_filter` / `until_filter`); - the presentations (*Instructors* below): `present_start` (the instructor's `login`, `host`, `rln`, `audience`: the number of logins and groups chosen), `present_audience` (the audience changed), `present_stop`, and `present_open` (a student opened the presented page: their `login`, the `presenter`); the project calls a spectator makes keep their own events, their close lines carrying `spectator=1`. The presence socket every page keeps open, the view the presenter publishes and the pointer frames of their presentation socket are not logged; - the focus on a student (*Instructors* below): `focus_start` (the instructor's `login`, `host`, `rln`, the `student`) and `focus_stop` (the `instructor` of the focus, the `student`); the view the two pages share and their sockets are not logged; - the web conferences (*Web conference* below): `webconf_start` (the instructor's `login`, `webconf` = the id of the room, its `name`, `audience` = the number of logins and groups), `webconf_audience` (the audience changed, the `owner` of the room), `webconf_attend` (an instructor joining a room), `webconf_stop`, and `webconf_join` (a join URL served to a participant: their `login`, `webconf`, `role`, status 302); the listing of the rooms is a keepalive, the page the client comes back to after *Leave meeting* is not logged; - the private conversations (*Private conversations* below): `webconf_invite` (the instructor's `login`, the `student` asked, the `host` and `rln` of the project they asked from, the `invitation`; status 409 when the student is not connected or busy), `webconf_accept` (the student's `login`, the `invitation`, the `instructor`, `webconf` = the room made; 410 when the invitation was over, 502 / 503 as the server answered), `webconf_decline`, `webconf_cancel` (the invitation taken back by an instructor, the `student`) and `webconf_leave` (one of the two ending the conversation: `webconf`, `owner`); an invitation nobody answers expires without a line. The client address is the last entry of `X-Forwarded-For` when the TCP peer is one of `trusted_proxies` (the Apache and nginx examples set that header), the peer address otherwise: set `trusted_proxies` in every mode to log the real addresses behind a proxy. The login rate limit (`[limits] login_per_minute`) counts per logged address too. The log holds personal data (names, addresses, what was done when): `keep` is its retention, and the file is readable by `sre-web` and root only (`/var/log/sre-web` is mode 0750). **In a database.** `[access_log.database]` sends the same events to a MariaDB / MySQL table through PyMySQL (`python3-pymysql`, suggested by the Debian package rather than installed with it since the feature is off by default, or `pip install pymysql` in the venv; the file and the journal stay what they are), where the History page of the administrators (*Administrators* below) reads them: ```toml [access_log.database] host = "db.example.org" # or unix_socket = "/run/mysqld/mysqld.sock" user = "sre_web" name = "sre" # the database; password = "…" in an imported secrets file keep = 400 # days of rows kept (purged daily); 0 = forever ``` One row per line: the time (`ts`, UTC, millisecond precision), the event, the login (empty when nobody was logged in), the client address, the host, the project (`rln`), the lab, the status and the duration in columns, the other fields of the line as JSON in `details` (`SELECT login, COUNT(*) FROM access_log WHERE event = 'start' GROUP BY login` works as expected). The table (`table`, `access_log` by default) is created when it is missing, so the account needs `CREATE` for the first start; or create it by hand and grant `SELECT, INSERT, DELETE` only (`DELETE` for the purge of `keep`): ```sql CREATE DATABASE sre CHARACTER SET utf8mb4; CREATE USER 'sre_web'@'%' IDENTIFIED BY '…'; GRANT SELECT, INSERT, DELETE ON sre.* TO 'sre_web'@'%'; CREATE TABLE sre.access_log ( id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT, ts DATETIME(3) NOT NULL, event VARCHAR(32) NOT NULL, login VARCHAR(64) NOT NULL DEFAULT '', addr VARCHAR(64) NOT NULL DEFAULT '', host VARCHAR(64) NOT NULL DEFAULT '', rln VARCHAR(255) NOT NULL DEFAULT '', lab VARCHAR(255) NOT NULL DEFAULT '', status VARCHAR(16) NOT NULL DEFAULT '', duration DOUBLE NULL, details TEXT NOT NULL, PRIMARY KEY (id), KEY ts (ts), KEY login_ts (login, ts), KEY addr_ts (addr, ts), KEY event_ts (event, ts) ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci; ``` The rows are written by a thread of the portal, in batches, and a request never waits for the server: while it does not answer, the rows wait in memory (`queue` of them at most, the oldest dropped beyond; one warning in the portal's log, one line when the writing resumes; the History page says it too) and the portal goes on; a batch the server refuses is written row by row and a refused row is dropped with a warning; the rows still waiting when the portal stops get one last attempt. The rows older than `keep` days are deleted once a day (`DELETE … LIMIT`: a replicated server needs a row-based binlog). Every consultation of the History page is a `history` line, in the file and in the table alike. The table holds the same personal data as the file: `keep` is its retention, and the account of the portal should be the only one reading it. In the production container the driver must be in the image (or the venv) and the server reachable from it (a local MariaDB over the host network, or its socket bind-mounted). **Retention.** The portal keeps `keep` days of each store while it runs: the rotated files beyond `keep` go at the next rotation, the rows older than `keep` days once a day. `scripts/web-purge-log.py` applies a retention of its own, for a cron job or a timer of the host (run with the Python of the portal, PyMySQL included; it reads `web.toml`): `--days` (365 by default), `--dry-run`, `--optimize` (the table rebuilt after the deletion, so that the deleted rows no longer occupy its file; a short lock), `--files-only` / `--database-only`, `--archives-days N` (the archives of the evaluations too, below; left alone otherwise); the current file is never touched, a rotated file goes by its modification time. For instance in `/etc/cron.d/sre-web`: ``` 15 4 * * * root /opt/sre/venv/bin/python3 /opt/sre/scripts/web-purge-log.py --days 365 ``` ### The archives of the evaluations Every `sre eval` writes an archive on the host that ran it (`params.archive_dirs`, `/var/lib/sre/archives`: the student's self-evaluations, the periodic evaluations of the web agent, an instructor's, the final one of a project closed with `eval_before_exit`, those of the Qt GUI), and the Watch page reads the newest archive of each *running* project. With `[access_log.database]` configured, the portal also collects **every archive of every pool host into a second table of that database** (`[archives]` of `web.toml`: `table`, `archives` by default; `keep`, `0` = the rows are kept forever; `poll_interval`, `30s`), and the **Archives** page of the instructors (*Instructors* below) reads it: the results of a project survive its closing, the history of its evaluations is kept, and the results of a student or of a class group can be searched across the labs and the hosts. The portal pulls, as it does for everything else: every `poll_interval` the collector asks each reachable agent for the archives of its host it has not taken yet (`GET /v1/archives?after=&limit=50`, the token alone: the archives whose name is greater than the last one taken, in the order of the names, which start with the evaluation time; up to ten calls per host and round, so that a host with a backlog of thousands of archives, the first run on a site that already evaluated for months, is imported at about a thousand archives per minute), and queues one row per archive. The agent decodes the archives and sends what the page needs: the project, the lab, the evaluation time, the student's login, full name, group, e-mail, the host that evaluated, the exam flag and fields, the exo-eval and self-eval totals and marks, the errors and warnings, and every element of the grade list with its title and description as the lab wrote them (in every language of the lab: the browser shows the viewer's), its grade, maximum, letter, part and bonus flag. **The answers of the students are not sent and not stored.** The group and the full name an archive lacks (the projects of the Qt GUI) are completed from the directory when the portal reads one (`attr_group`, `ldap_roles` / `ldap_groups`), as on the Watch page. The **reports** an instructor downloads from the page (the spreadsheet of `sre sheet`, the PDF reports of `sre outline` or their summary spreadsheet alone: *Instructors* below) are built from these rows, not from the archive files of the hosts, which may be asleep or reinstalled: a row holds everything the two commands read from an archive (the student, the time, the totals, the grade elements with their titles and parts, the number of errors; never the tests nor the answers). The portal turns the rows back into the records the agents had sent and hands them to **one** agent of the pool, chosen as the lab list is (an enabled, reachable host, up and little loaded first; the next one when it does not answer), which runs `sre report` (an internal command: the records on stdin, the file on stdout, one report at a time per host, `[timeouts] report` of `agent.toml`, `600` seconds) and answers the download the portal streams to the browser. The files are therefore those `sre sheet` / `sre outline` would give on the same archives, in the language of the instructor, but for one thing: a student is a login whatever the host of the pool the project ran on, where the two commands tell a login on two hosts apart (*Reports*, under *Instructors* below); the elements of an evaluation whose details were too big for the table (over 1 MB) are missing from them. The PDF reports and their summary read the grade elements of one evaluation per student only, the best one (the highest total, the earliest of them): when every evaluation of the selected projects is asked, the portal sends the others as *light* records (the student, the time, the totals, the error counts: about 300 bytes instead of several kilobytes), read without their details from the database, so that such a request may cover 150 000 evaluations (hundreds of projects evaluated every minute for hours) where the spreadsheet of `sre sheet`, which shows the elements of every evaluation, stops at 5000. Every download is a `report` line of the connection log (the kind, the rows ticked, the options, the evaluations and projects sent, `full` = the evaluations sent with their details, the host that built the file). One cursor per host, the greatest archive name taken, lives in the portal's memory and is read from the table itself at the first round after a start (`SELECT host, MAX(archive) … GROUP BY host`): nothing is collected while the database does not answer, and a restart never imports a host again. The name of an archive is unique in the table (`UNIQUE (archive)`, `INSERT IGNORE`), so a host whose archive directory is shared with another one, or rows still queued when the portal stopped, only cost a harmless re-insert. An archive rewritten in place (`sre re-eval -p '' -d `) or named below the cursor (a host whose clock was set back) is not taken again, as `sre watch` would not see it either: the results of a re-evaluation reach the table through `sre re-eval --update-web` (*Re-evaluated archives* below). An agent not yet updated answers 404 and is skipped with one warning until it is. The table is created like `access_log` when it is missing (the `CREATE` privilege, once); or create it by hand and grant `SELECT, INSERT, DELETE` on it (the same grant as the log's: the account needs no `UPDATE`): ```sql CREATE TABLE sre.archives ( id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT, ts DATETIME(3) NOT NULL, -- the evaluation time, UTC host VARCHAR(64) NOT NULL DEFAULT '', -- the pool host (the name of [[hosts]]) archive VARCHAR(300) NOT NULL, -- the file name: unique rln VARCHAR(255) NOT NULL DEFAULT '', -- the running lab name lab VARCHAR(255) NOT NULL DEFAULT '', -- sre/dhcp.py: the path the student started canonical_lab VARCHAR(255) NOT NULL DEFAULT '', -- the file really run, symlinks resolved (sre/dhcp.py for s3/r306/dhcp.py -> ../../sre/dhcp.py) login VARCHAR(64) NOT NULL DEFAULT '', fullname VARCHAR(255) NOT NULL DEFAULT '', grp VARCHAR(64) NOT NULL DEFAULT '', -- the student group hostname VARCHAR(64) NOT NULL DEFAULT '', -- the host the archive names grade DOUBLE NOT NULL DEFAULT 0, -- exo-eval scope max_grade DOUBLE NOT NULL DEFAULT 0, mark DOUBLE NULL, maximum_mark DOUBLE NULL, errors INT NOT NULL DEFAULT 0, warnings INT NOT NULL DEFAULT 0, exam TINYINT NOT NULL DEFAULT 0, auto_eval_count INT NULL, -- the self-evaluations of the student so far details MEDIUMTEXT NOT NULL, -- JSON: elements, parts, errors, self-eval totals, metadata PRIMARY KEY (id), UNIQUE KEY archive (archive), KEY ts (ts), KEY login_lab_id (login, lab, id), KEY login_canonical_id (login, canonical_lab, id), KEY rln_id (rln, id), KEY lab_ts (lab, ts), KEY canonical_lab_ts (canonical_lab, ts), KEY grp_ts (grp, ts), KEY host_archive (host, archive) ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci; ``` **Upgrading a table created before `canonical_lab`** (releases up to 0.15.0): the portal probes the columns when it connects and, while one is missing, keeps the rows in memory and warns once (`archives database: the table archives lacks a column the rows need …`) instead of refusing them; add the column and its keys from the root console before the portal restarts with the new code, then give the old rows their own path as the file run, and run that `UPDATE` once more after the restart: the rows the old code inserted meanwhile have an empty value (the account of the portal has neither `ALTER` nor `UPDATE`): ```sql ALTER TABLE sre.archives ADD COLUMN canonical_lab VARCHAR(255) NOT NULL DEFAULT '' AFTER lab, ADD KEY login_canonical_id (login, canonical_lab, id), ADD KEY canonical_lab_ts (canonical_lab, ts); UPDATE sre.archives SET canonical_lab = lab WHERE canonical_lab = ''; -- to merge the old rows of a symbolic link with their lab, by hand: -- UPDATE sre.archives SET canonical_lab = 'sre/dns1.py' WHERE lab = 's3/r306/dns1.py'; ``` The new rows carry the file really run as soon as the agents run the new code (an older agent sends none and the row gets its path). The rows go through the same kind of writer as the log's (batches, the rows waiting in memory while the server does not answer, a refused row dropped with a warning, a last attempt at shutdown) and are purged daily beyond `keep` days when set; `web-purge-log.py --archives-days N` applies a retention from a cron job. Expect about thirty rows per project and hour of auto-evaluation (`eval_interval_without_exam_mode`), a few kilobytes each. The table holds personal data (names, groups, results): the account of the portal should be the only one reading it, and the page is open to the instructors only. Every consultation of the Archives page is an `archives` line of the connection log, with the filters asked. ### Re-evaluated archives A grading bug found after the evaluations is fixed in the lab file and the archives are graded again with [`sre re-eval`](cli.md#post-exam-management), which replays the lab on the test results each archive holds. The table of the portal was filled when the evaluations ran and does not follow by itself; **`sre re-eval --update-web`** makes it follow: ```bash # as root, on the pool host whose /var/lib/sre/archives holds the archives sudo sre re-eval -s /opt/sre/lab/s3/r306/dns1.py -p corrected -d /root/regraded \ --update-web /var/lib/sre/archives/*@@@s3@r306@dns1.py@@@* ``` The command writes the re-graded archives as usual (`-p`, `-d`) and, for each of them, queues the new results on the host: one file per archive in `/var/lib/sre/web-updates/` (mode `0700`, the files `0600`, owned by `sre`: they name students and grades), holding what the agent sends for an archive — never the answers — under **the name of the original archive**, which is how the portal knows the evaluation. A later re-evaluation of the same archive replaces the one still waiting. Nothing else is to be configured: no address of the portal, no token, no privilege of the database account. The portal takes them as it takes the archives. The answer of `GET /v1/archives` says how many re-evaluations wait on the host (`updates`); when there are some, the collector reads them in pages (`GET /v1/archives/updates?after=&limit=`, the token alone), writes each row again in one transaction per page, and tells the agent which ones are settled (`POST /v1/archives/updates`), which removes them from the host. A row is written with `REPLACE` under its own `id` — the `INSERT` and `DELETE` privileges the account already has; no `UPDATE` — so it keeps its place on the Archives page and the links to it. Of the stored row it also keeps what only the first collection knew: | Kept from the stored row | Taken from the re-evaluation | |---|---| | `id`, `ts` (the time of the evaluation), `host` | `grade`, `max_grade`, `mark`, `maximum_mark` | | `canonical_lab` (the lab file as it resolved then) | `errors`, `warnings` and the error list | | `fullname`, `grp` (the directory's, at the time) | the elements and the parts of the grade list | | the `managed` detail | the self-eval totals, `re_eval_date` | The reports downloaded from the page are built from the rows, so they show the new results at once. The Watch page reads the archive files of the running projects, not the table. - **Only the evaluations the table holds are updated; none is added.** A re-evaluation whose evaluation is not there yet (collected at the next round, or the database did not answer) stays on the host and is tried again at every round; the agent gives it up, with a line of its log, after a day. Give the archives under their original names: the output of an earlier `re-eval` and a renamed archive are not queued. - **Run it where an agent runs.** The queue is read by the agent of the same host. With an archive directory shared by the pool (`archive_dirs`), one run on any pool host updates the evaluations of every host: a row is found by the name of its archive, whichever host it came from. On a host without `agent.toml` the command warns that nothing will hand the updates over. - **Applied at least once.** The portal acknowledges after its transaction is committed; a round interrupted in between writes the same row again at the next one, which changes nothing. A database or an agent failing here is warned about once in the portal's log (`archives: (re-evaluations): …`) and never stops the collection of the archives. - The portal logs `archives: : N evaluation(s) replaced by their re-evaluation`; the Archives page's database status counts them (`collector.updated`). A re-evaluation of an archive whose row belongs to another project is dropped with a warning. - An agent older than this feature never announces the queue and a portal older than it, or one without `[access_log.database]`, never asks for it: the files wait on the host. Whatever waited for more than a day is dropped when the queue is next read by the agent or written by `sre re-eval --update-web`, which then warns that the portal never took them. ### User groups The **Groups** page of the instructors (`/groups`, the link in the header) lists the groups of users the portal knows, of three kinds, each with its members (login, name, department): - **departments**: the values of `attr_group` of `[auth.ldap]` (`departmentNumber`) over every person of `search_base`, the student group the portal has always carried per user (the Group column of the Watch page), listed as groups; - **directory groups**: the entries of `group_search_base` matching `groups_filter` (`posixGroup` by default), named by `attr_group_name` (`cn`), their members the values of `attr_group_members` (`memberUid`): each value is looked up among the people as their login, their `uid` or their DN (a `groupOfNames` with `member` DNs works with `attr_group_members = "member"`); a value matching nobody but shaped like a login counts as that login, the DN of a nested group is dropped; - **custom groups**: the groups the instructors create on the page (a name of at most 64 characters without `:` nor `/`, which may not be that of a department or of a directory group, a description), fill (logins typed or picked among the people of the directory, which refuses an unknown login; or every member of another group of any kind at once), empty (one member at a time, or at once every member who also belongs to another group of any kind) and delete. Any instructor may change any custom group; the creator and the dates are shown. A user may belong to several groups. The directory is read whole, with the service account (or anonymously) of `[auth.ldap]`: two paged searches of `PAGE = 500` entries (the people base of a site exceeds the default size limit of a server), kept `directory_ttl` and read again on demand (the *Refresh* button, at most once per `refresh_interval`: the clicks arriving during a read get that reading, the next ones within the interval the reading just made, so that a row of clicks never dumps the directory once per click). The listing needs `uri`, `search_base` and the login attribute of the people: `attr_login`, or the attribute of a `search_filter` of the form `(attribute={login})` (`(supannAliasLogin={login})`); the departments need `attr_group`, the directory groups `group_search_base`. It works in every authentication mode, proxy mode included. The page says what is not configured, and shows the error of a directory read that failed (the previous reading stays until the next attempt, one minute later). The custom groups live in two tables of the database of `[access_log.database]` (`[groups] table`, `user_groups` by default, and `
_members`), on iff that section is set. The tables are created when they are missing (the `CREATE` privilege, once), or create them by hand; the account needs `SELECT, INSERT, UPDATE, DELETE` on these two tables (`UPDATE`: a renamed group or a changed description; the other tables keep `SELECT, INSERT, DELETE`): ```sql CREATE TABLE sre.user_groups ( id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT, name VARCHAR(64) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL, description VARCHAR(255) NOT NULL DEFAULT '', created_by VARCHAR(64) NOT NULL DEFAULT '', created_at DATETIME(3) NOT NULL, PRIMARY KEY (id), UNIQUE KEY name (name) ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci; CREATE TABLE sre.user_groups_members ( group_id BIGINT UNSIGNED NOT NULL, login VARCHAR(64) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL, added_by VARCHAR(64) NOT NULL DEFAULT '', added_at DATETIME(3) NOT NULL, PRIMARY KEY (group_id, login), KEY login (login) ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci; GRANT SELECT, INSERT, UPDATE, DELETE ON sre.user_groups TO 'sre_web'@'%'; GRANT SELECT, INSERT, UPDATE, DELETE ON sre.user_groups_members TO 'sre_web'@'%'; ``` The `name` and `login` columns are binary-collated: the names and logins are exact and case-sensitive, like the logins everywhere else (a table created with the default collation would find `INFO1` for `info1`). The members of a deleted group are deleted with it, in one transaction; the rows are never purged. Every group is named by a key `kind:name` (`department:1A`, `directory:etudiants`, `custom:TP lundi`; a bare name is a department: the links and audiences made before the kinds keep working). The groups of every kind are offered by the group select of the Watch page (the column still shows the department) and by the audience dialog of a presentation: a student is in the audience when one of their groups is chosen, the department by the group of their identity, the others by the memberships the portal knows, read again every 30 s and after every change made on the page (a student added to a presented group is told at once, one removed loses the presentation). Every reading of the page is a `groups` line of the connection log, every change a `group_edit` line (`action` = `create` / `update` / `delete` / `add` / `remove`, `group`, and `member` (the one member removed), `source` (the group whose members were added or removed), `count` (the logins given to an add, the members a removal by source took out) or `new_name`). ### Exams The **Exams** page of the instructors (`/exams`, the link in the header) programs the exams of the web site. An exam is a title, an **audience** (logins and/or groups of any kind of the Groups page: a department, a directory group, a custom group), the **labs** to do (the exam-only labs, whose names hold `_EXAM_`, `_OLD_`, `_DRAFT_` or `_TESTS_`, are offered here and nowhere else; each lab may name a preset of its `Flavor` class, as `sre set-exam --labs lab:preset` does), a **duration** and/or a **window** (`start_after`, `end_before`; no start time = the exam starts when created, for each student as soon as they log in), two delays before the start — the **lead** (the other projects of the students are hidden and a countdown appears) and the **pre-start** (their exam projects are started in the background) —, the **networks** the students' browsers must come from (none = anywhere; the address is the client's, through `trusted_proxies`) and the **evaluation interval** of the agents during the exam. The defaults of the delays and of the interval are those of `[exams]`. Unlike the exam mode of the Qt GUI (`sre set-exam`, `exam.json`, which locks a whole workstation and keeps a pool host out of the pool), an exam of the web site binds **its students only**, wherever their projects run: the portal computes the phase of each student and enforces it on every request, the agents apply the exam rules to the exam projects alone. Instructors are never bound, even when listed in the audience. What a student sees, in order: - **scheduled**: nothing changes before the lead window. - **waiting** (the lead window, 10 min by default): a bar on every page counts down to the start; their other projects are hidden and refused (the Open page too); from the pre-start (5 min before) their Projects page starts the exam projects in the background, one at a time (the start slot and the placement of a normal start; a lab asking a flavor form shows it), and shows them without an Open link. - **starting**: the start passed (or the exam has no start time), the exam projects are being started; the clock of the student starts when every lab of theirs has a running project. - **active**: the countdown to their end; the Projects page lists the exam projects alone (a single one is opened by itself at the start); the project page has no Close button, no Evaluation tab nor Apply Configuration tab (as in the Qt GUI); the agent evaluates the project every `eval_interval` of the exam. Each student has an individual clock: their end is `min(started + duration, end_before)` plus the minutes an instructor adds to them; an *End now* ends everybody's exam, or a selection's, at once. - **ended**: the end banner (*That's all folks*) until the student presses *Continue*; the agent has cut the terminals and the X desktop, run the final evaluation and refuses every change (the answers included). After *Continue* the usual pages are back; the exam projects stay hidden from the student (and refused) until they are closed: `close_after` after the end by the agent (the idle timeout of an exam project counts from its end), or by the instructor (*Close projects* on the page, or the Watch page; the page stops them in the background, eight at a time, and shows the progress under the title of the exam; a second click while a close runs changes nothing). - A student connecting from an address outside the networks of their exam gets the bar in red naming their address, and no project at all (neither the exam's nor the others) while the exam binds them. A student in several exams at once follows the one started first. The phases are pushed to the open pages through the presence socket (`{"t": "exam", "id", "phase"}`: the pages read `GET /api/exam` then) and the countdowns use the portal's time. On the page the instructor sees, per exam, every student the portal can name (the logins, the members of the groups it knows, those it has seen), connected or not, with their address, their phase, their clock, their extra minutes and their running exam projects (a link to each: the instructor's view of the student's project, the Evaluation tab included), and can, for everybody or for a selection (the checked rows): add or take minutes (for everybody the duration and the end time move together and an *End now* is lifted; for a selection their extra minutes, cumulative), *End now*, change the labs (the browsers start the new ones; the projects of a lab dropped stay as plain projects), *Close projects*, *Start now* (the start time set to now), *Edit* (every field; the students keep their clocks and adjustments) and *Delete* (the projects become plain projects, or are closed with the checkbox). The listing shows the scheduled, the running and the past exams. The exams live in two tables of the database of `[access_log.database]` (`[exams] table`, `exams` by default, and `
_members`), on iff that section is set (the page says so otherwise): an exam survives a restart of the portal, the clocks of the students included, and the agents are told again what they missed. The tables are created when they are missing (the `CREATE` privilege, once), or create them by hand; the account needs `SELECT, INSERT, UPDATE, DELETE` on the exams table (an edited exam) and `SELECT, INSERT, DELETE` on the members one (its rows are replaced): ```sql CREATE TABLE sre.exams ( id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT, title VARCHAR(200) NOT NULL, created_by VARCHAR(64) NOT NULL DEFAULT '', created_at DATETIME(3) NOT NULL, updated_at DATETIME(3) NOT NULL, start_after DATETIME(3) NULL, end_before DATETIME(3) NULL, duration INT NULL, ended_at DATETIME(3) NULL, details MEDIUMTEXT NOT NULL, PRIMARY KEY (id), KEY start_after (start_after), KEY created_at (created_at) ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci; CREATE TABLE sre.exams_members ( exam_id BIGINT UNSIGNED NOT NULL, login VARCHAR(64) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL, first_seen DATETIME(3) NULL, started_at DATETIME(3) NULL, ended_at DATETIME(3) NULL, dismissed_at DATETIME(3) NULL, extra_minutes INT NOT NULL DEFAULT 0, address VARCHAR(64) NOT NULL DEFAULT '', details TEXT NOT NULL, updated_at DATETIME(3) NOT NULL, PRIMARY KEY (exam_id, login) ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci; GRANT SELECT, INSERT, UPDATE, DELETE ON sre.exams TO 'sre_web'@'%'; GRANT SELECT, INSERT, DELETE ON sre.exams_members TO 'sre_web'@'%'; ``` `duration` is in seconds; `details` holds the audience, the labs, the delays, the networks and the interval (JSON), the members' `details` their labs and their exam projects. The times are UTC, truncated to the millisecond like the rows of the connection log. The past exams are kept `keep` days (`0` = forever), their members with them. The agents: an exam project is started by the portal with an `exam` object (`POST /v1/projects`: the id and title of the exam, the start and end of the student's clock once known, the evaluation interval) and a privileged `sre start` run under the student's login (`LOGNAME`), which accepts the exam-only labs and the flavor presets (`--set-flavor-name`); the agent keeps the object in the project's state file and in the rows of its listings (`exam`), evaluates the project at the interval of the exam once its clock runs, never reaps it before the end, refuses the owner's self-evaluation, states and close during the exam and every change after the end, and at the end cuts the terminals and the display, runs the final evaluation (`sre --user eval`: the exo-eval scope, like `sre end-exam`) and tells the open pages (`{"t": "exam", "ended": true}` on the events socket). The portal updates the object with `PUT /v1/projects/{rln}/exam` when the clock starts or the instructor adjusts the student (every 5 s until the agent has it), and drops it with `DELETE …/exam` when the exam is deleted or the lab is no longer the student's. The answers metadata of an exam project carry the exam fields of the Qt GUI's exam mode (`exam_mode`, `exam_started_at`, `exam_duration`, `exam_time_remaining`), so the archives carry the exam flag and the Archives page filters on it. The exam-only labs of the pool are listed by `GET /v1/labs?all=1` (instructors), a privileged `sre list --with-titles --include-exam-only`. Connection log: every change of an exam is an `exam_edit` line (`action` = `create` / `update` / `delete` / `end` / `adjust` / `labs` / `close`, `exam`, `count` of students, `minutes`), a student dismissing the end banner an `exam_dismiss` line, and the `start` lines of the exam projects carry `exam`; the polls of the page and of the student's exam are keepalives. Limitations: the session records (`record_sessions`) are not archived by the agents (the lab's attribute decides whether `records/` exists, `sre stop` deletes it); the archives of a web exam hold no `exam_json` document (the `exam` flag comes from the answers); anonymous mode has no stable logins to schedule; a page loaded before the portal was updated has no exam script until it is reloaded (the server still refuses what it must). ### Web conference (BigBlueButton) An instructor talks to a student, a group or the whole class without leaving the site: the portal opens **rooms** on a BigBlueButton (BBB) server and shows its client to the participants in an iframe over the pages of the site, while everybody goes on working on their projects. Nothing is installed on the pool hosts; the portal needs the API of the server, the browsers the server itself. ```toml [bbb] url = "https://bbb.example.org/" # the server (its bigbluebutton/ or bigbluebutton/api/ path works too) secret = "..." # the shared secret of its API: bbb-conf --secret # checksum = "sha1" # sha1 (default), sha256 or sha512: supportedChecksumAlgorithms of the server # welcome = { fr = "Bienvenue", en = "Welcome" } # the welcome message of the client (the server's by default) # record = false # mute_on_start = false # duration = 0 # "2h": the server ends a meeting after that; 0 = unlimited # max_participants = 0 # poll_interval = "10s" # the participants and the ended meetings are read that often # private_rooms = true # Talk privately on a student's project (Private conversations below) # [bbb.userdata] # the client settings of every join (userdata-bbb_*, the BBB API reference) # bbb_skip_check_audio = true # [bbb.private_userdata] # the client settings of the joins of a private conversation, over the others # bbb_listen_only_mode = true ``` **The instructors**: the *Webconf* button of the site bar opens the dialog of the rooms: those open on the portal, with their owner, the size of their audience, who is connected to the site and who is in the room (read from the server every `poll_interval`), each with *Join* (this browser shows the room, as a moderator: any instructor may join any room), *Audience…* and *End* (any instructor); and a *New room* form: a name (the instructor's by default) and an audience chosen as for a presentation (the connected users one by one; the groups: departments, directory groups, custom groups; a member of a group connecting later is put in the room too). Unlike that of a presentation, the audience of a room may be empty (*None* then *Choose*): the room is then one of the instructors alone. A student is in at most one room at a time (the newest one including them); an instructor sees the room they created or joined last. **The participants**: the client appears in a floating window over the page (dragged by its title bar, resized by its corner), with buttons to minimize it to its title bar, to dock it into the **Webconf** tab of the project page, to show it full page (a bar above it comes back), to pop it out to a separate browser window, and to leave (×). How it is shown and where the window is are remembered by the browser: the next page shows the conference the same way (in a window where there is no project tab). A participant who left, or whose client is in a separate window, keeps a pill in the site bar to rejoin or to bring it back; the *Leave meeting* of the client itself leads to that pill. Every page of the site loads the client again (a new join of the same room: a few seconds, the audio to join again), which a site of several pages cannot avoid: the separate window does. Two tabs of the same user never join twice: one holds the client, the others show a pill to bring it there. **The server**: the meeting is created when the room is (a server that refuses or does not answer leaves no room: the dialog says so); a join is a URL signed for the user (`fullName` = their name, `userID` = their login, the role by their instructor flag: a moderator for an instructor, a viewer otherwise, the `bbb.userdata` settings), served by the portal as a redirect (`GET /api/webconfs/{id}/join`, the `src` of the iframe): the secret never leaves the portal. A meeting nobody joined expires on the server after a few minutes (`meetingExpireIfNoUserJoinedInMinutes`): the portal creates it again at the next join. A meeting everybody left expires about a minute later (`meetingExpireWhenLastUserLeftInMinutes`), like one a moderator ended from the client: the room ends on the portal at its next poll, every participant told. *End* in the dialog ends the meeting on the server. The rooms live in memory: a portal restart forgets them (the meetings expire on their own, the clients already in them stay until then). **Prerequisites**: the portal reached over HTTPS, with `public_url` set (the `logoutURL` of the meetings, where the client comes back after *Leave meeting*, is built from it: `/webconf/left`); the server reachable from the portal (its API) and from the browsers (the client, over HTTPS); the nginx of the server must not forbid framing by the portal's origin (an `X-Frame-Options` header or a `frame-ancestors` directive on the client would leave the frame empty: the separate window is the fallback); the pages of the portal allow the server's origin in the `frame-src` of their `Content-Security-Policy`, nothing else. BBB 2.4 at least (the `role` of the join; the passwords are still passed for older servers). Safari may refuse the camera in a cross-origin frame: pop the client out. #### Private conversations An instructor who opened a student's project (the helper view) may talk with that student alone, voice and text. BigBlueButton has no private audio inside a meeting (one audio bridge per meeting; its breakout rooms are made by hand in the client and open in another browser tab), so a private conversation is a **second meeting of the two**, which both browsers show *in place of* the class conference for as long as it lasts: nobody is ever in two meetings at once. - **Talk privately**, in the bar of the project after *Focus student on project* (shown with `[bbb]` unless `private_rooms = false`), invites the owner of the project. Nothing else happens yet: both stay in the class conference, no meeting is created. The button reads *Calling… (cancel)* and takes the invitation back; an invitation nobody answers expires after a minute. The student must be connected to the site, and neither invited by somebody else nor in a private conversation already (the instructor is told so). - **The student** gets a banner across the top of every page of the site they have open (above a presentation, inside an open dialog), with a short chime and a mark in the title of the page: the name of the instructor, *would like to talk with you*, and two buttons, *Join* and *Not now*. *Not now* tells the instructor; nothing opens a microphone before *Join*. - **Join** creates the meeting and sends both browsers to it: the window of the conference (at the place and in the mode of the class conference: window, tab or full page) now shows the private room, titled *Private —* and the name of the other one, its title bar in its own colour. The joins go straight to the microphone, without the echo test or the choice of *Listen only* (`bbb_auto_join_audio`, `bbb_listen_only_mode = false`, `bbb_skip_check_audio`, `bbb_hide_presentation_on_join`; `[bbb.private_userdata]` overrides them, e.g. `bbb_listen_only_mode = true` to get the choice back); the meeting is never recorded and starts nobody muted, whatever `[bbb]` says for the rooms. The chat of that meeting is the text channel. - **Only the tab where the user clicked** — *Talk privately* for the instructor, *Join* for the student — loads the client: the other tabs, and any other browser the same user is logged in from, leave the class conference and show the pill (*Bring here* moves the conversation). A browser left open somewhere never opens its microphone by itself. - **The end**: the × of the window, *End conversation* in the bar of the project, or the *Leave meeting* of the client, by either of the two — leaving a conversation of two ends it; *End* in the Webconf dialog, by any instructor (the dialog lists the conversation, nobody else may join it or see its audience); and the server itself, a minute after the instructor is gone (a closed browser: `endWhenNoModerator`). Both browsers then show the class conference again, in the mode and at the place it had; an instructor comes back to the room they were in. - An instructor already talking with a student may invite another one: the first conversation ends when the second one is accepted, and the instructor goes from one to the other. Joining another room from the Webconf dialog ends the instructor's conversation. Limits: each switch reloads the client (a few seconds, twice per conversation), and both are out of the class conference meanwhile — the other students cannot reach the instructor there. A class conference popped out to a separate window is closed when the conversation starts (by the tab that opened it) and comes back in a window of the page; when that tab was closed since, the site cannot close the separate window: the user is then in both meetings until they close it. The invitations live in memory like the rooms. The connection log gets the `webconf_*` events (*The connection log* above). Tests: `tests/test_bbb_client.py`, `tests/test_web_webconf.py` (the registry on a fake client, the portal in front of the fake server of `tests/fake_bbb.py`), `tests/test_web_webconf_browser.py` (headless Chrome: the rooms, then the private conversations — the banner, the switch and the return of two pages, two tabs of one user holding one client). ### Securing the pool (mutual TLS) The portal and the agents can authenticate each other with certificates of one private CA, on top of the token: the agent serves TLS and requires a client certificate of that CA, the portal verifies the agents against the same CA and presents its client certificate. `scripts/web-tls.sh` makes the files with `openssl`: ``` scripts/web-tls.sh init-ca /root/sre-tls # ca.key, ca.crt (10 years) scripts/web-tls.sh agent /root/sre-tls a01 IP:10.0.0.11 DNS:a01.example.org # a01.key, a01.crt scripts/web-tls.sh portal /root/sre-tls # portal.key, portal.crt ``` On each pool host, `agent.toml` gets its own certificate and the CA (`tls_cert = ".../a01.crt"`, `tls_key = ".../a01.key"`, `tls_ca = ".../ca.crt"`, files readable by root only) and, as a second fence, `allowed_clients = ["
"]`. On the portal host, `web.toml` gets `[pool] ca = ".../ca.crt"`, `client_cert = ".../portal.crt"`, `client_key = ".../portal.key"` (readable by `sre-web`), and every `[[hosts]]` URL becomes `https://:8451`. The portal then logs `TLS to the agents: CA …, client certificate …` and each host as reachable; an agent without the right certificate, or a client without one, fails at the handshake, before any request. ### A whole site at once (`scripts/make-web-hosts.sh`) `scripts/make-web-hosts.sh` writes everything the previous section asks for, for every pool host of a site, from a list of its machines: ``` IP_MAC_FILE=ip_mac.txt OUT_DIR=/root/sre-pool REGEXP='^([a-z])[0-9]+$' ROOMS="a h" DISABLED_ROOMS=h \ scripts/make-web-hosts.sh ``` The list has one `mac;ip;name` line per machine (`e4:54:e8:d1:6c:1c;10.0.128.1;a01`: the MAC address for Wake-on-LAN, the IPv4 address the portal reaches its agent at, the name). A machine gets an agent when its lower-cased name matches `REGEXP`; the first group of the expression, when there is one, is its room, so that `ROOMS` selects the rooms joining the pool (empty: every qualified machine) and `DISABLED_ROOMS` those whose hosts start with `enabled = false` (polled, no project placed there until `sre-web-tool enable`). These five variables are required and have no default; `AGENT_PORT` (8451), `MAX_PROJECTS` (25), `POWER` (`power = true` in the entries and `allow_power_off = true` on the agents), `DOMAIN` (a `DNS:.` SAN too), `PORTAL_ADDRESS` (`allowed_clients` of the agents), `INSTRUCTOR_ROOMS` (the rooms whose hosts get `instructors_only = true`) and `INSTRUCTOR_MACHINES` (the names of the hosts that get it on their own, whatever their room), `TLS_DIR_ON_HOSTS` (`/etc/sreweb/tls`, the paths written in the files) and `CA_KEY` / `CA_CRT` (both set: an existing CA copied into `OUT_DIR` instead of a new one, so that the hosts of a new `OUT_DIR` are signed by the CA the portal already knows; refused when `OUT_DIR` holds another CA) are optional. In `OUT_DIR` (private keys and tokens: outside any repository) it writes, through `web-tls.sh`, the CA and the portal's client certificate once, then per host a key and a certificate (SANs `IP:` and `DNS:`), a token in `.token`, and: - `hosts.toml`: the `[[hosts]]` entries, one room after the other, with the three `[pool]` TLS lines of `web.toml` in its header; a file of `web.d/` (`import = ["web.d"]`) or pasted at the end of `web.toml`; - `agent-.toml`: the host-specific lines of that host's `agent.toml` (`listen = ":8451"`, `token`, `tls_*`, and `allowed_clients` / `allow_power_off` when asked), to complete with the example file, whose own `listen` and `token` lines are then removed (a key given twice is refused). Copy `.crt`, `.key` and `ca.crt` to `TLS_DIR_ON_HOSTS` of each host, and `ca.crt`, `portal.crt`, `portal.key` to the portal's. Every run computes `hosts.toml` and the agent files from the variables of the moment, so the rooms, the disabled or reserved hosts may change at any time, and compares them with the previous run: a file whose content did not change is left as it is, the others are replaced, what changed is printed per host (`enabled`, `instructors_only`, `allow_power_off`...) with the new hosts and those no longer selected, and the last line says whether `hosts.toml` and which agent files are to deploy again; the CA and the portal's certificate are made once. The hosts' keys, certificates and tokens are made anew on every run, to deploy again, unless the script is run with `--keep`: the files already there are kept and only those of the new hosts are made (a kept certificate that no longer names the address of its host is reported), so a room is added, or a host disabled or reserved, without touching the hosts already deployed. ### A pool of workstations The pool hosts are typically classroom workstations: off when nobody needs them (a nightly cron shuts them down), woken by Wake-on-LAN, and sometimes used at the keyboard by a student. The portal gives each host a **state** from the status of its agent, and `sre-web-tool list` (or the *Pool* page of the instructors) shows it: | State | Meaning | New projects | |---|---|---| | `off` | the agent does not answer | no | | `waking` | a wake was sent less than `wake_timeout` ago | no, but a start answers "a host is starting" (503, retried by the Open page) | | `up` | the agent answers, the host is free | yes | | `occupied` | the agent answers, the host is used locally (`occupied_check`) | only the new projects of the users logged in there (`follow_local_sessions`); the web projects already running there go on | | `exam` | the agent answers, `exam.json` is on the host | no | | `draining` | the portal asked the host to power off, or `sre-web-tool drain` | no; powered off once it is empty | | `disabled` | `sre-web-tool disable`, or `enabled = false` | no; the projects already running there go on (listed, usable, evaluated and closed as usual), the host stays polled and is never powered off by the portal — `drain` is the way to empty a host and shut it down | **Placement.** A new project (`POST /api/projects`) goes to one host, chosen by the portal from the last status poll of every agent (`poll_interval`) without asking them again. The candidates are the hosts `up` (plus, for a user logged in at the keyboard of an `occupied` host, that host: *Occupied hosts* below), below their `max_projects`, not reserved for the instructors when the caller is a student, and whose agent has no restart pending (`sre-web-tool restart`: the agent refuses new projects until it is back). Among them: the host of the user's own session first, then the lowest `projects / max_projects`, then the lowest one-minute load, then the name, so that equal hosts are always settled the same way. The project count of the chosen host is raised at once and a *reservation* is kept until the start is over and a status poll sent after its end has answered: a burst of simultaneous starts is spread over the pool instead of piling onto the host the last poll showed emptiest, and `sre-web-tool list` may count a project one poll before the agent does. Without any candidate the start is refused with 503 and a `retry_after` the Open page obeys: "a host is starting" (30 s) when a host the caller may use is `waking` or restarting, "no host available for a new project" (60 s) otherwise. An instructor naming a host on the Open page skips the ranking: the host must exist (404), be open to them (403 for a student), be available to them (503 "not available") and have room (503 "full"). The `start` line of the connection log says why the host was chosen: `placement=load`, `session` or `named`. **Waking.** A host with a `mac` gets a magic packet (UDP port 9 to `wol_broadcast`: the portal must be on the hosts' network, or the routers must forward the directed broadcast); a host with a `wake_command` gets that command instead (`ssh relay wakeonlan {mac}`, an IPMI or VM call…). The portal wakes one sleeping host, in configuration order, whenever the free project slots of the hosts up (plus the capacity of those waking) drop below `spare_projects`, and keeps `min_hosts_up` hosts awake during `active_hours`: the floor is re-established in the morning and not fought at night after the cron. A host still silent after `wake_timeout` is retried after 1, 2, 4… up to 30 minutes, so a machine that is really down is not hammered. The agent must be an enabled systemd unit on every host (boot → agent up → the portal sees it). **Occupied hosts.** The agent reports `occupied` when a local login session exists (`loginctl`, `who` as a fallback) or when projects of the Qt GUI run on the host; the portal places nothing there and never powers it off. The web projects already running there continue. One exception, `follow_local_sessions` (`true` by default): the user logged in at the keyboard of a pool host gets their new projects on that host, whatever its load, so that a student working at a classroom workstation uses their own machine through the web too. The agent names the sessions in its status (`session:`, the OCCUPIED column of `sre-web-tool list`; its `occupied_check` must include the sessions) and the portal matches them against the login of the identity, which must therefore be the Unix login of the workstation (the directory `uid`: `AuthLDAPRemoteUserAttribute uid` in the Apache example; a `guest-…` pseudonym never matches). The rule overrides the load and the occupation only: when the host is full (`max_projects`), in exam mode, disabled, draining or reserved for the instructors, the normal placement applies, and an instructor naming a host on the Open page is obeyed (they may name the host of their own session). The `start` line of the connection log says `placement=session`. `follow_local_sessions = false` restores the strict rule: nothing goes to an occupied host. **Reserved hosts.** A host with `instructors_only = true` never takes a student's project: an instructor's goes there when they name it on the Open page (the *Host* select lists the reserved hosts with their state and their projects) or by the automatic placement when it is the least loaded host they may use. It is outside the elastic pool: its free slots count neither as spare capacity nor towards `min_hosts_up`, the portal never wakes it by itself (`sre-web-tool wake`, or its own boot; an instructor finds it greyed while it is off) and powers it off when idle like any host with `power = true`, whatever the floor and the spare. `list` shows `(instructors)` after its state. A student logged in at the keyboard of a reserved host is still placed elsewhere: `follow_local_sessions` never overrides the reservation. **Going down.** When the agent's unit stops because the system halts (the nightly cron, a `poweroff`), the agent closes the projects it manages as a Close would (the last evaluation, then `sre stop`), concurrently, within `TimeoutStopSec`; the browsers see "project closed". The unit is ordered after `remote-fs.target` and, on a checkout, requires the mount of `/opt/sre` (`RequiresMountsFor`): it is stopped while `sre` and the archive directories are still mounted. Unordered, systemd tries to unmount `/opt/sre` while the agent is still closing a project: the unmount fails (`device is busy`) and is not tried again, the network goes down with the NFS mount still there, and the machine never powers off (it then answers neither ping nor Wake-on-LAN until somebody presses its power button). What is left on the host afterwards (the projects of the Qt GUI, a project not closed in time) is removed by the unit `sre-wipe-at-shutdown.service` when it is enabled (`sre wipe` after the agent has stopped and before Docker does; see *Stopping the projects when the machine halts* in the [installation](installation.md) page), so that no container is running when the system powers off. With `power = true` on the host and `allow_power_off = true` in its `agent.toml`, the portal powers off an idle host itself: `idle_host_timeout` (0 by default: only the cron does it), above `min_hosts_up` and as long as the spare capacity stays, or at once with `sre-web-tool poweroff`. **Pre-loading the images.** A project whose Docker images are not on its host yet waits for their download at its start, and a room of workstations pulling them at the same time takes the whole bandwidth of the site; the boot unit `sre-preload-images.service` (see [Installation](installation.md)) only spreads the pulls over a random delay. On the pool, the agents pull and the portal lets **a few hosts at a time** do it: - a host *asks*: its agent starts with `[preload] at_start = true` in `agent.toml` (the boot of the host, a restart of the agent), or `sre-web-tool preload NAME | --all | --failed` or the *Preload images* button of the Pool page ask for it by hand (after new labs or a new image version). The agent then says `pending` in its status and pulls nothing; - the portal *gives turns*: at every poll, as many waiting hosts as `[pool] max_preloads` (2) leaves free get `POST /v1/preload/start`, the one that waited longest first; the agent then runs `sre preload-images --progress` on its lab directories (`[preload] paths`) and reports the image it pulls. A request by hand goes through the same queue: the number of hosts downloading never exceeds `max_preloads`, whoever asked; - every host that asks takes a turn, even with every image already there: its `docker pull`s only check the registry, a few seconds, and an image pushed again under the same tag is fetched. `sre-web-tool list` shows `(preload queued)`, `(preload 2/5)` or `(preload failed)` after the state, the Pool page the same as flags (the image being pulled, or the error, as tooltip), and `sre-web-tool status` the count of each. A preload that fails (`failed`: an image that does not exist, the registry unreachable; the error names the images) is not tried again by itself: `sre-web-tool preload --failed` asks again. A disabled or occupied host is served like the others — its local users start labs too —; a host is not given a turn while projects of an exam of the web site run on it, while it is draining or its agent restarting, and an idle host waiting for its turn is not powered off by `idle_host_timeout`. A stop or a restart of the agent stops the preload in progress (Docker keeps the layers already there): with `at_start` the agent started next asks again, a request made by hand is forgotten. Nothing is preloaded without the portal (the hosts stay `pending`; the images are then pulled by the first start that needs them, as without any preload) nor with `max_preloads = 0`; the agents older than this feature are left alone. On a host whose agent has `at_start`, leave `sre-preload-images.service` disabled: both would pull. Nothing asks again on a timer. A host that boots every day checks at each boot; one that stays up does not notice new labs or a new image version until its agent restarts (`sre-web-tool restart --all` after a sync makes every `at_start` agent ask again) or a preload is asked by hand, and a failed one stays failed. A cron job on the portal's host makes the check periodic, through the same queue (`max_preloads` hosts at a time): ``` 0 3 * * * sre-web-tool preload --all ``` **Restarting the agents.** After an update of the tree the agents run the old Python code until their unit is restarted: `sre-web-tool restart NAME`, or `restart --all` for every reachable host (whatever its state), tells each agent to restart itself (`POST /v1/restart`). The agent answers at once, lets its running `sre` commands end (a start, an evaluation; no new project is accepted meanwhile), then runs `systemctl reset-failed` and `systemctl --no-block restart` on its own unit (`sre-agent` or `sysreseval-agent`, read from `/proc/self/cgroup`; `restart_command` in `agent.toml` replaces that for an agent run otherwise). The projects stay (`close_projects_on_stop = "shutdown"`: a restart is not a shutdown; with `always` they are closed), the open terminals and virtual desktops are lost, the project pages reconnect by themselves. The stop is quick: the agent closes the project pages' WebSockets itself (aiohttp would otherwise wait a minute for them) and `KillMode=mixed` ends what is left. `list` shows `(restart)` after the state until the portal sees the agent's uptime reset (or gives up after two minutes with a warning in its journal); the host takes no project meanwhile and keeps its drain. `--wait` makes the tool wait for the agents to be back (`--timeout`, 180 s by default), so a deployment script can chain it after the sync. A `DO_NOT_START_AGENT` marker written since the start is honoured (that agent stops); code that no longer starts leaves the unit failed after five attempts (`journalctl -u sre-agent`, then `systemctl reset-failed`). The portal itself is restarted by hand (`systemctl restart sre-web`; *Restarting the portal* below). **By hand.** `sre-web-tool …` (`sbin/sre-web-tool` of a checkout, `/usr/sbin/sre-web-tool` of the `sysreseval-web` package; `python3 -m sreweb.tool` from the sources) talks to the running portal through its administration API (`[admin] token`, loopback only by default; `--config`, `--url`, `--token` override the configuration): ``` sre-web-tool list # one line per host: state, projects/max, load, terminals, occupied by, uptime sre-web-tool status # spare capacity, hosts awake vs the floor, active hours, hosts waking, the state file sre-web-tool add a03 --url https://10.0.0.13:8451 --token … [--mac … | --wake-command …] [--max-projects 25] [--power] [--instructors-only] sre-web-tool remove a03 [--force] # refused while the host has projects (--force leaves them to the agent's idle reaper) sre-web-tool disable a03 | enable a03 # disable: no new project on the host, the running ones go on until their users close them sre-web-tool wake a03 | poweroff a03 | drain a03 # drain: no new project, powered off once the last project is closed sre-web-tool restart a03 | --all [--wait] [--timeout 180] # the agents restart themselves (new code), the projects stay sre-web-tool preload a03 | --all | --failed # the hosts pull the Docker images of the labs, max_preloads of them at a time ``` `list` shows `config` or `added` per host; a config host removed by hand comes back, enabled, with `enable`. `--json` gives the raw rows. The instructors have the same actions on the **Pool** page of the portal (`/pool`, see *Instructors* below), except `add` and `remove`, which stay with the token. **Restarting the portal.** What is changed by hand, with the tool or on the Pool page, is kept in `[admin] state_file` (`pool.json`, written at every change through a temporary file renamed over it) and merged with the `[[hosts]]` of `web.toml` at every start, so that a restart of the portal leaves the pool as it was: | Kept | How | |---|---| | the hosts added and removed by hand | `added` (the whole entry, its token included: the file is `0600`), `removed` | | a host enabled or disabled by hand | `enabled` / `disabled`: the hosts whose flag differs from the `enabled` of their `[[hosts]]` entry, in both directions (a host of `DISABLED_ROOMS` enabled for a class stays enabled) | | a drain | `drain`, with the time it was asked; over when the host is gone, or was booted again while the portal was stopped | | a wake in progress, the pause after failed wakes or after a power off, a power off or a restart sent | `clocks` (written within one poll and when the portal stops); each is dropped at the start when too old to mean anything | The file only holds what differs from the configuration, and **an edit of the configuration wins**: a flag set by hand holds for as long as the `enabled` of that host in `web.toml` is what it was then. Once the file says the same (the room taken out of `DISABLED_ROOMS`, for instance) the entry is dropped at the next start, and a later edit of `web.toml` is followed again; so is the entry of a host that is no longer in the pool (one line of the journal each). The start logs `pool state read from …: 0 removed, 0 added, 21 enabled and 1 disabled by hand, 0 draining`. A state file that cannot be written (a read-only directory, a container without its volume) would lose all of this at the next restart: the portal says so in its journal at the start and at every failed write (`pool state NOT saved in …`), `sre-web-tool status` prints `pool state: NOT SAVED in …` instead of the path, and the Pool page shows it in red above its table. In a container, `/var/lib/sre-web` must be a mounted directory the `sre-web` user may write; the systemd units provide it (`StateDirectory`). A file the portal cannot read (edited as another user, or no longer a JSON object) is not written over: it is renamed `pool.json.unreadable`, said in the journal, and the pool starts from the configuration; edit the file with the portal stopped and leave it to the `sre-web` user. Not kept: the idle clock of `idle_host_timeout` (it counts again from the start), and what is listed under *Limitations* (the presentations, the focuses, the open sockets: the pages reconnect by themselves, a terminal with its Reconnect button). The exams, the custom groups, the connection log and the archives are in the database; the sessions are in the browsers' cookies (`secret`). ## What the users see - **Header**: the SRE logo and, with `[site]` in `web.toml`, the institution's logo and name next to it, on every page. - **Login**: the directory account, or (anonymous mode) a display name and a language; in proxy mode the web server in front has already asked, there is no login page. - **Projects**: the user's running projects on the whole pool (Open, Close), "Open a lab" (the lab tree with its titles, in the order the `titles.json` of each directory sets, by file name, by title in the language of the page or by hand, the sub-directories first, the same tree as the Qt GUI's; the Flavor form when the lab has one; the progress of the pulls and deploys); with no running project the page shows the SRE logo in large, as the Qt GUI does when no project is open. An instructor's cards say the mode of each project (Standard / Instructor mode / Debug mode, the words of the Open page), the two non-standard modes also as a coloured suffix after the title. Logout asks what to do with the running projects: close them (each as a Close does: the last evaluation when the lab asks for it, then `sre stop`; the dialog waits for every answer) or leave them running; with no running project the button logs out at once. - **A project**: the tabs of the Qt GUI. The schema is drawn by the agent (graphviz) with the same options (lines, spacing, order, icons); the questions and the forms are the same; the evaluation table has the same parts, subtotals, bonuses and colours; the terminals are `xterm.js` terminals on `sre connect`: every Connect button of the Machines and Networks tabs opens a new one on its device in a **floating window** over the page (like the external xterm of the Qt GUI: `gw`, `gw (2)`, …; the window stays while the tabs change, is dragged by its title bar, resized by its bottom-right corner and closed by its ×, and takes the font size and the colour scheme of the Terminals tab), the two selects of the Terminals tab open one on any device (the machines open to the students and the managed switches) as a sub-tab or in a window, the ⧉ of a sub-tab moves its terminal into a window and the dock button of a window brings it back as a sub-tab (the same shell goes on), the × of a sub-tab or of a window closes it, `terminals_per_device` caps them; the Log tab appears for debug projects. *Export Schema to PDF* downloads the drawing of the Schema tab with its current options (never the hidden machines), rendered by the agent as the GUI's File → Export Schema to PDF does, unless the lab sets `export_schema_pdf = False`; the icons need `rsvg-convert` (`librsvg2-bin`) on the pool host, graphviz shapes otherwise. The Kathara export is not offered on the web. A terminal is a session shared by its viewers: the live sessions of the project that are not on the page are listed under the bar with a Join button (the same shell for everybody, the last 64 KiB of output replayed to the newcomer, the pty sized to the smallest window), a sub-tab or a window shows how many viewers its session has, and a session ends when the shell exits or when its last viewer closes it with its ×. A viewer that just goes away (the page closed, reloaded or cut, another project opened) leaves the session running for `terminals_linger` of the agent (120 minutes by default), listed *idle* meanwhile, and the page coming back to the project finds the user's own terminals again, sub-tabs and windows where they were on that browser, with the last 64 KiB of output (a page the browser kept in its back/forward cache, brought back by the Back button, joins its sessions again by itself); after a disconnection, Reconnect joins the same session again. - **A presentation**: when an instructor presents a project to the class (*Present* below), the page of every chosen student is covered by that project, read-only: every tab is there but nothing can be done on it, and the tab shown, the terminals open (the instructor's shells, in their exact size, at the largest font that fits; the instructor's floating windows float on the student's page too, at the same relative place and size), the question selected, the drawing options of the schema, the instructor texts and the virtual desktop (view-only) follow what the instructor does; so does the position inside the tab: the scroll of the text, of the question (its description and the list of titles), of the tables and of the log (the line at the top of the instructor's pane is at the top of the student's, whatever the size of their window or their font size), and the pan and zoom of the schema. The instructor's mouse pointer is drawn on the student's page too, an arrow carrying the instructor's name, and what the instructor selects — in the texts, the question, the tables, the log, the schema, or in a terminal — is highlighted there (in a terminal: selected in the student's read-only terminal, in lavender); both are placed on the element, the word or the terminal cell the instructor's mouse is on, never at the same pixel, so that a student with another window size, font size or word wrap sees the pointer on the same word. The Evaluation tab shows the table of the instructor's page as it is (the letters of their own self-evaluation included), again after each evaluation. The title bar of the presented project is coloured (lavender) and names the instructor presenting it. The student's own page is untouched underneath, terminals included: when the presentation ends it is back as it was. ### Instructors Who is an instructor is decided by the directory, the role being the OR of the tests configured in `web.toml` (a test whose key is empty is skipped): - `instructor_groups`: the user is a member of one of these groups (DNs), through the `memberOf` attribute of their entry or, with `group_filter = "(member={dn})"`, a search of the groups under `group_search_base` (`search_base` without it); - `instructor_filter`: the user's own entry matches this LDAP filter (a BASE search on their DN), `"(gidNumber=2000)"` by default — the primary group; a site with the memberof overlay may write `"(memberOf=cn=teachers,ou=groups,dc=example,dc=org)"`; `""` disables the test; - `group_search_base` + `instructor_group_filter`: a group found under that base names the user (`{login}` and `{dn}` are substituted, filter-escaped), by default a `posixGroup` of gid 2000 listing the login in `memberUid` (the group search that needs no overlay; a `groupOfNames` site writes `"(&(cn=teachers)(member={dn}))"`); setting `group_search_base` enables the test. In LDAP mode the portal runs them when the user logs in (the role is kept in the session cookie: a change applies at the next login). In proxy mode the reverse proxy authenticates and the tests are run on the request headers (`instructor_groups` on the groups header, `instructor_values` on `instructor_header`: any attribute `mod_authnz_ldap` exports) and, with `ldap_roles = true`, by the portal itself with the `[auth.ldap]` settings for the login the proxy gives (the service account or an anonymous search finds the entry, then the same three tests; the entry's `attr_group` is read at the same time, so with that key set the lookup runs for the users the headers already made instructors too, who keep the role whatever the directory says), the result being cached per login for `ldap_roles_ttl`; a directory failure makes the login a student without group for a minute (logged once) rather than the site unavailable. `ldap_groups = true` does the group part alone: the directory gives the group of a user whose header carries none, the roles stay those of the headers (the production case: `gidNumber` exported by Apache, `departmentNumber` read by the portal for the owners of the Qt GUI projects on the Watch page). What an instructor gets: - **Watch** (the link in the header, `/watch`): every running project of every reachable host, one row per project with its host, owner (the login), name (the full name of the answers metadata, or the directory's for a project of the Qt GUI when the portal reads it, above), the owner's student group when the site configures one (`attr_group` / `header_group`), lab, start time, the **last action** of its owner (how long ago the student last did something on the project: a key pressed, a click or a turn of the wheel anywhere on their project page, which reports them on its events socket at most every 10 s — a tab change, a question picked, the course read, the virtual desktop —, and what the agent sees by itself, to the second: the input of a terminal, an answer, an evaluation, a state; a mouse that only moves is no action. The owner's actions only: what an instructor helping on the project types and what the spectators of a presentation do never count, so the column tells whether the student is at work. The agent keeps the time across its restarts; the cell is blank for a project the student uses from the Qt GUI only, and for a host whose agent is older than the column), the age of its last archived evaluation, its grade and mark (exo-eval scope, as `sre watch` shows them), its errors and warnings, the number of self-evaluations the student has run (the agent counts the lines of the project's `.private/auto_eval.log`, one per `sre eval --auto-eval` of the student: live, whereas the `auto_eval_count` of an archive, the AUTO-EVAL column of `sre watch` and the Self-evaluations column of the Archives page, is the count before that evaluation), its open terminals and whether the web or the Qt GUI runs it; sortable by any column, filtered by text (host, owner, group, lab) and, with a group source, by the select listing the groups by kind (the departments of the rows, the directory groups and the custom groups of the Groups page, *User groups* above; every group, one group, or *No group*: the projects whose owner has no department, those started on the Qt GUI carrying none unless the portal reads the directory, above), the chosen group kept in the URL (`?group=kind:name`, a bookmark per class group) and remembered by the browser (`localStorage`): the page comes back with the group chosen last, through the *Watch* link of the header or in a new tab, until another group or *All groups* is chosen, a link with `?group=` winning and becoming the remembered one; refreshed every 5 s; a row whose last evaluation had errors is red, one without an evaluation younger than 15 minutes (or twice the lab's evaluation interval) is amber. Open goes to the project, Close stops it. The agents read the archives of their own host (`params.archive_dirs`) on demand, nothing is scanned while nobody watches. As in `sre watch`, the page also shows the details: a **click on a row** opens a dialog with the last archived evaluation of the project (the owner, the lab and the host in the title, the date of the evaluation, the table of its grade elements with the parts, the subtotals, the bonuses, the total and the mark, then the errors and the warnings of the run; *Open the project* goes to its page; Esc or *Close* goes back), and a **click on the lab cell** of a row opens the summary of that lab for the rows shown (the text filter and the group select apply, so a class group alone when wanted; the lab is the file really run, so the projects started through a course directory's symbolic link to a lab and through the lab's own path are one lab): a matrix with one row per project and one column per grade element (numbered, the element as the tooltip of the number, the parts as a header row; a cell is the grade, or the OK / MEH / FAIL letter, coloured like the evaluation table; Total and Mark last; a project never evaluated says so), the rows *Max*, *Average* and *Success* (how many of the evaluated projects got the full grade) below it, then the table of the elements with their maximum, minimum, average and distribution of the grades (`1.5(2) 3(7)`), a subtotal row per part, as the lab summary of `sre watch`. The elements of the projects are matched by their title, whatever the language of each student. A **click on the group cell** of a row (when it holds a group) opens the same summary for that lab restricted to the whole group: every project of the lab on the pool whose owner has that group, whatever the text filter (the lab cell keeps the rows shown; an empty group cell opens the last evaluation like the other cells). Both dialogs follow the polls: a new evaluation of the project, or of one of the projects of the lab, reloads the content, a project closed is said so. Reading the results of a student this way never counts as their activity (their project's idle clock is left alone). - **Archives** (the link in the header, `/archives`): the evaluations collected into the database (*The archives of the evaluations* above; without `[access_log.database]` the page says so), the closed projects included. The **Evaluations** view lists them newest first, one row per evaluation with its time, the student (login, name, group when a group source is configured), the lab, the pool host, the start of the project, the grade and the mark (exo-eval scope), the errors / warnings counts, how many self-evaluations the student had run and whether it was an exam; a row whose evaluation had errors is red. The form searches by *Student* (the beginning of a login, or a part of a name), *Group* (a group of the table, with suggestions), *Lab* (a part of its name, with suggestions), *Host*, a *From* / *To* window on the evaluation time and the exam flag, and chooses what to *Show*: **the last evaluation of each project** (the default: the final result of every project, among the evaluations matching the window, the host and the exam flag), **the last evaluation of each student** (one row per student and lab) or **every evaluation** (the progression of a project). *Rows* and the *Older* / *Newer* buttons page through the result; a click on a login, group, lab or host cell filters on that value; the filters and the view are kept in the URL (a search can be bookmarked or shared; `?open=` opens one evaluation: a link to a result). A **click on a row** opens the evaluation as the Watch page's dialog does (the student, the lab and the host in the title, the time of the evaluation and of the start of the project, the table of its grade elements with the parts, the subtotals, the bonuses, the total and the mark, laid out by the browser from the stored elements exactly as the agent lays out the evaluation of a running project, then the errors and the warnings of the run). The **Summary** view (the second button of the toolbar; a lab is required: without one the view lists the labs of the archives, a click on one loads its summary; the search starts on the last evaluation of each student) is the lab summary of the Watch page for the evaluations found: one block per lab matched (the lab file really run, as on the Watch page: the evaluations of the projects started through a symbolic link to a lab are in the lab's block, and the lab filter matches the path started as well as the file), with the matrix of the students and the grade elements (Total and Mark last, the rows *Max* / *Average* / *Success* below) and the table of the elements with their statistics; a click on a row of the matrix opens the evaluation. The summary is computed on the 500 newest evaluations found at most (the page says so: narrow the filters). Searching the results of a class group is `Group` + `Lab` + *Summary*; the results of one student across the labs, `Student` alone. **Reports.** A checkbox at the left of each row selects evaluations (the header checkbox takes the rows shown; the selection is kept while paging and searching, the bar says how many are selected and how many are not on the page), and the bar under the filters downloads a **Spreadsheet** (what `sre sheet` gives: per lab, the table of every evaluation, the *Questions* statistics and the *Sessions* best grades), the **PDF reports** (what `sre outline` gives: a zip holding one PDF per student and lab, `__.pdf`, and `outline.ods`, the summary) or the **Summary spreadsheet** (what `sre outline -o` gives: that `outline.ods` alone, one row per student and lab with the best evaluation, its time and the mark) of the selected evaluations. **A student is a login here, whatever the host**: the portal places each project on any host of the pool, so the instances of a lab a student opened several times are merged into one row, one PDF and one *Sessions* row even when they ran on different hosts (the two commands, run by hand on the archives of a host, keep one per login *and* hostname: the hostname is the student's workstation there, and tells apart the users of a shared account); the `hostname` of a merged row, and of the header of its PDF, is the host of the best evaluation, and each history table of the PDF is headed by the start time and the host of its instance. A selected evaluation stands for its whole project: every stored evaluation of the same projects goes into the reports (the history table of a PDF, every row of the spreadsheet, the best evaluation of the summary), as `sre sheet archives/` would do on a host; the *Options* disclosure unticks that (**Every evaluation of the selected projects**: the ticked rows alone then) and gives the flags of the two commands: **One report / row per project instance** (`--separate-instances`: a student who opened a lab twice gets two reports, or two summary rows, instead of one with two history tables), **"Bonus" before the label** (`--bonus-in-prefix`), **Evaluation history**, **Remaining time column** (exams) and **Group by part** (the PDF only: the two spreadsheets ignore them). At most 1000 rows may be ticked; the PDF reports and the summary spreadsheet may cover 150 000 evaluations (300 projects of 300 evaluations give their PDFs in about a minute), the spreadsheet 5000 (the page says so: narrow the selection or untick the option). The file is named after the lab when the selection is of one lab (`evaluations_dhcp_20261109-1430.ods`, `reports_dhcp_20261109-1430.zip`, `summary_dhcp_20261109-1430.ods`), is built by one agent of the pool from the stored rows (*The archives of the evaluations* above) in the language of the page, and is saved by the browser; the buttons read *Generating…* meanwhile, a few seconds for a class. - **Pool** (the link in the header, `/pool`): every host of the pool, one row per host with its state as `sre-web-tool list` shows it (the seconds and the failed attempts of a wake), its projects and terminals, its load, who occupies it, its uptime, its flags (reserved for instructors, power, drain, power off sent, restart sent, and where the preload of the Docker images is: *preload queued*, *preloading 2/5* with the image as tooltip, *preload failed* with the error), the agent's version and its last error, the line of `sre-web-tool status` above the table (in red, with the reason, when the state file of the pool cannot be written: a restart of the portal would undo what is changed on the page); the state cell is green (up), amber (waking, occupied, exam, draining), red (off: the agent does not answer) or grey (disabled, muted text); sortable by any column, filtered by host name (one or several words: the hosts whose name contains any of them, `a01 a02 h` for those two and every `h…` host), refreshed every 5 s. The buttons of a row are the commands of the tool: *Enable* (green) / *Disable* (red), *Wake* (light green; a host with a `mac` or a `wake_command`, while it is off), *Drain*, *Power off* (asked to confirm; allowed by the agent's `allow_power_off`, refused while the host has a project or a local session), each disabled with the reason as its tooltip when it does not apply. A checkbox per row (the one of the header ticks the hosts shown by the filter) selects hosts for the bar above the table: the count of the selected hosts (and how many the filter hides), the same five buttons and *Preload images* (the bar only: the selected hosts wait for their turn to pull the Docker images of the labs, see *Pre-loading the images* above), each acting on every selected host its action applies to, one call per host run together (the tooltip says how many, the button is disabled when none; *Power off* asks once with the names; the failures come in one message, the selection stays). Adding and removing hosts, and restarting the agents (`restart`, shown as the flag *restart sent* while it lasts), stay with `sre-web-tool`. Every action, of a row or of the bar, is an `admin` line of the connection log with the instructor's login. - **Projects** stays the instructor's own projects (Watch holds the others). - **A student's project**: the student named in the project bar after the lab title (full name and login, from the answers metadata; the login alone when the file has no name); every tab, the devices all open (the privileged `sre connect`), the answers editable (what the instructor types goes into the student's answers file, whose metadata stays the student's: the Watch page and the archives still name the student; every save of either page is pushed to the open pages of the project, so the student's page shows the instructor's text within a second and the instructor's page the student's); the top of the page (the site bar and the project bar) takes the **helper colour** (teal) on the instructor's browser only, a reminder that the project is not theirs (the student's page stays grey; the presentation colour replaces it while the project is presented); the Terminals tab **joins the student's live terminals** when it opens (the instructor sees the student's shell and may type in it, the student sees the same; a closed sub-tab is not joined again), and the selects offer a **private terminal** (the student does not see it in their list; what the instructor types there stays with them) or a shared one, the Connect buttons a private one in a window; a terminal on a device the student cannot open themself (a closed machine or switch, a hidden machine) is **always private**, whatever the choice: the shared kind is greyed for it and the agent makes the session private anyway; the Evaluation tab opens on the **last archived evaluation** of the project (date, errors in the tooltip), shows each new one by itself (the periodic evaluations of the lab, the student's own, another instructor's: the agent tells the open pages) and *Start evaluation* runs a privileged `sre eval` (no cooldown, nothing counted on the student's side, the exo-eval marks); the states apply through the privileged path too. All this is for somebody else's project: on their own, an instructor is the student (see *Instructor mode* below). - **Focus student on project**: the toggle button of a student's project page (off by default) brings the student to what the instructor is looking at. While it is on, every page of the student that does not show the project goes to it (now, and any page they open later: the student stays on the project until the button is turned off), the top of both pages (the site bar and the project bar) takes the **focus colour** (a deeper teal than the helper colour, which comes back on the instructor's page when the mode is off), the student's page names the instructor following them, and the two pages share the **tab** shown, the **terminal** last gone to, sub-tab or floating window (a terminal the student may see: their own sessions and the shared ones, joined by both: a terminal one page shows is joined by the other, as a sub-tab, when it has not got it; a private terminal of the instructor is never shown nor named to the student, whose page keeps its sub-tab while the instructor is on one; a page brought to the Terminals tab by the focus opens no terminal of its own, as it does when its user clicks the tab for the first time) and the **question** selected, in both directions: whichever page changes one of them, the other follows, the last change winning. The student's page also shows the instructor's **mouse pointer** (an arrow with their name) and what the instructor **selects**, as the pages of a presentation show the presenter's: on the same paragraph, table cell or tab button, on the same row and column of a terminal both pages have, whatever the sizes of the two windows (nothing is drawn while the student is on another tab). A selected text is highlighted without touching the student's own selection; a selection in a terminal becomes the selection of the student's terminal. One way only: the instructor does not see the student's mouse. Nothing is sent of a private terminal of the instructor, nor of a place in a text the instructor reads with the instructor fragments shown (the *Instructor mode* button on the Informations, Questions and Apply Configuration tabs: the student's page has not got the same text). Unlike a presentation, the student keeps every control: they type in the terminals, change tab, open terminals, answer questions (and the instructor may write in the answers too, both pages showing the same text). The button's tooltip says whether the student's page follows. A reloaded instructor's page goes on focusing; closing it for a minute, closing the project, or turning the button off releases the student (their page stays where it is, grey again). Any instructor may end a focus; a student is in one focus at a time (the newest wins, an older one still running comes back when it ends). - **Instructor mode** on the Open page: the instructor's own project started with `sre start --instructor-mode` (the instructor texts kept in `info.json`, every state listed, the states logged): the *Instructor mode* button of the project page then shows the `instructor()` texts of the Informations and Questions tabs, every state and the Log tab, as in the Qt GUI. A student's project holds no instructor text (the CLI strips them), so the button does not appear there. The Evaluation tab of the instructor's own project is the student's one, as in the Qt GUI: *Start evaluation* runs the self-evaluation (`sre --user eval --auto-eval`, the cooldown of the lab, the letters OK / MEH / FAIL when the lab hides the marks with `no_mark_on_self_grade`) and the table it gives stays; the last archived evaluation is shown on somebody else's project only (an archive holds the numbers). - **Debug mode** on the Open page: the third of the *Project mode* radios of the instructors (*Standard*, *Instructor mode*, *Debug mode*). The instructor's own project is started with `sre start --debug-web-project`: the debug project of `sre start --debug-project` (see *Lab Authoring*) with the lab given by its name, so that it keeps the lab name of the catalog and is grouped with the lab's other projects on the Watch and Archives pages. While *Debug mode* is chosen the lab tree lists every lab of the host, the exam-only and draft ones included (`GET /api/labs?all=1`, as the Exams page does; the selection is kept when the lab is still listed). The project page then shows what the Qt GUI shows for a debug project: every machine, the hidden ones in red, each with a Connect button, the *Full Schema* tab (the hidden machines drawn while the *Instructor mode* button is on), every state, the Log tab with the states and the evaluations, the evaluation table with both scopes, no self-evaluation cooldown. `POST /api/projects` takes `instructor_mode` or `debug_project` (both together is accepted); a student sending either gets 403. Presenting a debug project shows its hidden machines to the spectators, who see the presenter's page. - **Groups** (the link in the header, `/groups`): the user groups of the site by kind (the departments, the groups of the directory, the custom groups), their members, and the editing of the custom groups (*User groups* above). - **Exams** (the link in the header, `/exams`): the exams of the web site, programmed for an audience of logins and/or groups, followed and adjusted while they run (*Exams* above). - **The host of a project**: the *Host* select of the Open page (shown when the pool has a host reserved for instructors, `instructors_only = true` in `[[hosts]]`) names one of them, with its state and its projects (greyed when off or full), or leaves the automatic placement: the least loaded host the instructor may use, reserved or shared. `POST /api/projects` takes the optional `host`; a student sending one gets 403. - **Present**: the *Present* button of a project page (the instructor's own project, or a student's) opens the list of the users connected to the site (a text filter, the groups of every kind with the number of their members connected: the departments, the directory groups and the custom groups of the Groups page, *Select all*): the chosen students, one by one and by group (a group takes its members connected now and those connecting later), see the project read-only on their own browsers (above). The top of the page (the site bar, the project bar and the bar under it) then takes the presentation colour (lavender), a reminder that the students are watching, and that bar tells how many students are in the audience and how many are watching; *Edit audience* changes the list, *Stop presenting* ends it (so does closing the project) and the bars are grey again. The spectators follow the instructor's page: the tab, the terminals the instructor opens (joined read-only: the students see the same shell, cannot type, and do not change its size), the question, the schema options, the *Instructor mode* texts, the Display tab view-only, the scroll of the tab shown and the pan and zoom of the schema (followed as the instructor scrolls, a few times a second), the instructor's mouse pointer (an arrow with their name, sent on the presentation socket about 25 times a second while it moves over the panes, the tab bar or a terminal; nothing is shown while it is elsewhere) and the text the instructor selects (in the page, highlighted without touching the student's own selection; in a terminal, selected in the student's read-only terminal), and the Evaluation tab shows them the table of the instructor's page as it is, sent through the portal after each evaluation, in the instructor's language (the archive of the project holds numbers only, while the instructor's own self-evaluation may be letters). A reloaded page goes on presenting; a page closed for a minute ends the presentation. A student is in one presentation at a time (the newest that includes them), and falls back to the previous one when it ends. - **Webconf**: the button of the site bar (when `[bbb]` of `web.toml` names a BigBlueButton server, *Web conference* above) opens the rooms: start one for chosen students and groups, join a colleague's as a moderator, change its audience, end it. The students put in a room get the client in a window over their page. - **Talk privately**: on a student's project, next to *Focus student on project*: the student is invited to a conversation of the two alone, voice and text (a banner, *Join* / *Not now*); once they join, both browsers show a private room instead of the class conference, and come back to it when the conversation ends (*Private conversations* above). ### Administrators An administrator is an instructor who may read the connection log on the **History** page (the link in the header, `/history`). Who is one is decided in `web.toml`, by login or by group, and a user who is not an instructor never gets the rights whatever the lists say: - `[auth] administrators`: a list of logins, in every mode; - in LDAP mode, `administrator_groups` of `[auth.ldap]` (DNs, tested like `instructor_groups`: the `memberOf` attribute, or the `group_filter` search) and `administrator_group_filter` (a group found under `group_search_base` naming the user, like `instructor_group_filter`; no default), run at login for the users the instructor tests accepted (the flag is kept in the session cookie, like the role); - in proxy mode, `administrator_groups` of `[auth.proxy]` on the groups header and, with `ldap_roles` or `ldap_groups` (a directory is configured either way), the two `[auth.ldap]` tests above, run by the portal for the instructors nothing else made administrators (cached per login like the roles; the flag is lost while the directory is down, like the role of `ldap_roles`). The History page needs `[access_log.database]` (*The connection log* above; without it the page says so) and has three views, chosen by the *Sessions* / *Events* / *Statistics* buttons of its toolbar. The page opens on the **Sessions** view: the log as one row per session (*The connection log* above: one login from one client address, from its first request to the logout, the idle timeout or the shutdown): the sessions still open first, in green, from the portal's memory (listed even without a database), then the closed ones, newest first, each built from its `session_close` line: login, name, student group (the column is shown when a group source is configured, `attr_group` or `header_group`, like the Group column of the Watch page), address, start, end (the last request: an idle session ends there, not at its expiry), duration, number of requests, the labs of the projects used (the running lab names as a tooltip) and what closed it (*Logout*, *Idle*, *Shutdown*; *Still open* with the number of pages, terminals and desktops connected). A click on a header sorts the rows by that column, a second click reverses the order (the default is the end time, newest first, an open session counting as not ended yet, hence first); the sort applies to the rows shown, the page, not to the whole log: widen the page with *Rows* or narrow the filters. Only the login and address prefixes and the time window apply; the window bounds the time a session was **closed** (`session_timeout` after the last request for an idle one; an open session is listed when it was active in the window); the other filters are hidden. The *show* cell of a row opens the events view on that session: its login and address, from the minute of its start to the minute after its closing line. The sessions closed before the name, role and group were logged on the closing line show an empty name and group. The **Events** view (`?view=events` in the URL) lists the events newest first, one row per event with its time (the local time of the browser), event, login, client address, host, project, lab, status, duration and the other fields of the line as `key=value`; a `denied` or `login_failed` line, a status of 400 or more and a failed start are red. The headers sort the rows shown the same way (the time, newest first, by default). The form filters by login and address (prefixes: `10.0.1.` is a subnet), event (every event name; in proxy mode without `login` and `login_failed`, which never occur there), host (exact), project and lab (substrings; the lab is only on the `start` lines) and a time window. The **Statistics** view (`?view=statistics`) says who connected over a period and for how long. The period is the *From* / *To* window of the form: switching to the view with an empty *From* fills it with the time 24 hours ago, and an empty *To* is now (the default period is the last 24 hours; the login and address prefixes apply, an address prefix being a room). A session counts when it was active during the period and only its part inside the period is counted; the time of a user is the union of their sessions (two devices at once are not counted twice); the sessions still open count up to their last request, the administrator's own among them. A user whose role is not *instructor* is a student (the sessions closed before the role was logged on the closing line count as students). The page shows the **totals** (users, students, instructors, groups, sessions, client addresses, the total connection time, the average, median and longest connection time of a student, the most users at once and when, the projects started, the projects used, the evaluations asked and the time spent on the project pages), the **groups** of the students (students, sessions, time, average, projects, evaluations; shown when a group source is configured, like the Group column; a click on a group lists the users of that group alone, a second click every user again), the **users** (name, group, role, sessions, connected time, time on the project pages, first and last request in the period, addresses, projects, evaluations, requests; sortable by the headers), the **labs** (one row per lab, every instance merged: projects, users, students, the time spent on the project pages and its average per user, the starts, the evaluations) and the **distribution** of the students' connection time (less than 15 minutes, 15 to 30, 30 to 60, 1 to 2 hours, 2 to 4, more). The time on a project page is that of its `project_close` lines (the page open as long as its events WebSocket was), clipped to the period and merged per user and lab; a page still open is not counted yet, and the pages of the spectators of a presentation never are; the starts are the `start` lines with `result=created`, the evaluations the `eval` lines answered 200; the requests, projects and evaluations of a session are those of the whole session. The portal reads the lines of those four events from the start of the period to two days after its end (a session or a page active in the period but closed later than that is missed), 200 000 of them at most: beyond, the page says the statistics are computed on the newest rows only and asks to narrow the period. The **Group** select of the form (shown when a source of user groups is configured, *User groups* above: the departments, the directory groups and the custom groups, listed by kind as `GET /api/groups` gives them) restricts the three views to the members of one group, whatever its kind: the portal resolves the group into the logins of its current members and asks the database for their rows alone (one `IN` list on the indexed `login` column, so the sessions, the events or the statistics of a class group are one query; the open sessions are filtered the same way). A group nobody belongs to as far as the portal knows (a custom group emptied or deleted since, a department of nobody) gives nothing and the page says so. The department cell of a session filters on that department. The filter is kept in the URL as `group=kind:name` (a bare name is a department, like the Watch page's) and logged as `group_filter`; without any group source the parameter is refused (400). In the two list views the filters stay in the URL (the sort of the headers does not), so a filtered view reloads and can be sent to a colleague; *Older* / *Newer* page through the rows (100, 500 or 1000 at a time, by a cursor on the row ids: a page is stable while new events arrive; each page is sorted on its own) and a click on a login, address or, in the events view, event, host or project cell filters on that value. Nothing is polled: *Search* asks again. Each query is a `history` line of the log itself, with the filters asked (`view=sessions` for the sessions, `view=statistics` and the period resolved for the statistics). In proxy mode (the production site behind Apache) the page shows the same sessions: a session starts at the first request carrying the proxy's identity headers, right after the browser's password dialog (no `login` line: Apache authenticates, the portal only sees the result); the Logout button still closes it on the portal before the browser is sent to `logout_url`, where Apache's 401 makes it forget the Basic credentials; a tab closed without a logout ends it by the idle timeout like the other modes; the credentials the browser cached make a later return a new session without any dialog; the client address comes from `X-Forwarded-For` of the `trusted_proxies` peer (set in every mode: the proxy's address alone would merge the devices of a user); the authentications Apache refuses never reach the portal (its own log holds them). ### The Display tab A lab machine declared `x11_host` draws its graphical applications (Wireshark, Firefox…) on the X server of the host: on a workstation, the student's own desktop. A pool host has no X server, so the agent runs one virtual desktop per project whose lab has such a machine: a TigerVNC `Xvnc` with an `openbox` window manager, started as the unprivileged `sreweb-x11` user right after `sre start`, shown in the **Display** tab by noVNC (`make novnc`). The containers reach display `:N` on TCP port 6000+N of the host with the cookie of the project (the core passes them `SRE_DISPLAY` and `SRE_XAUTH_COOKIE`, the `.bashrc` of the images ≥ 1.31 sets `DISPLAY` from them); the browser reaches the RFB Unix socket of the server through the agent and the portal, as a WebSocket of binary frames. The desktop follows the size of the tab (*Resize the desktop*; *Fit to window* scales it instead), the text box of the bar sends text to the X clipboard and, on the owner's page only, shows what the applications copy (put into the browser's clipboard too; an instructor helping on the project can send text but receives none: any X client of the containers can set the selection), *Reconnect* opens the stream again. The number and the cookie of a project never change: a desktop that died, or that was lost when the agent restarted (its X server is a child of the agent), is started again at the next use of the tab, empty. `[display]` of `agent.toml` holds the geometry, the range of display numbers and the commands. When no number of the range is free (every one taken by a project, by the lock file of another X server in `/tmp` or by a listener on its TCP port, which any local user can leave), a project still starts, without a desktop (no Display tab); the agent logs the reason once. Session recording (`asciinema`) applies as on a workstation, since it happens in `sre connect`. - The answers are saved as the user types (debounced), with the identity of the session. ## Security - The agent refuses every request without its bearer token and the identity headers the portal adds (`X-SRE-Login`, `X-SRE-Fullname`, `X-SRE-Group`, …); the browser never talks to it. With mutual TLS (above) only the holder of a client certificate of the CA reaches the token check, the traffic to the agents is encrypted, and `allowed_clients` refuses every other address with 403. Without TLS the token and the terminals cross the network in clear: keep the agents on a trusted management network or behind a tunnel (the agent warns at startup). - The session cookie is signed (HMAC-SHA256), `HttpOnly`, `SameSite=Strict`, `Secure` behind HTTPS; it lives 14 days from the sign-in (a language change sets it again with its original date: a session never slides) and carries the role, group and administrator flag of the sign-in (a change in the directory applies at the next sign-in: Logout, or the expiry); the state-changing calls need the `X-Requested-With: sre-web` header and the WebSockets a matching `Origin`; the pages carry a `Content-Security-Policy` (the origin of the BigBlueButton server added to its `frame-src` when `[bbb]` is set: the only other origin a page may frame). - In proxy mode nothing is stored in a cookie: every request carries the identity set by the reverse proxy, accepted from the peers of `trusted_proxies` only (a request reaching the portal from anywhere else has no identity, headers or not). - The directory is reached over TLS (`ldaps://`, or `start_tls` on `ldap://`) and the portal verifies the server certificate on every connection: the chain against the system CA store or `[auth.ldap] tls_ca`, the host of the `uri` against the names of the certificate; a StartTLS the server declines aborts the login, and a plain `ldap://` is refused at startup unless `allow_cleartext = true` (the student and instructor passwords, in LDAP mode, and the answers the roles and groups are built from would otherwise be exposed to anyone on the path). The `ldap3` library validates nothing by itself. - One account is one user: in LDAP mode the identity carries the login as the directory stores it (the login attribute of the entry found, else the typed login lower-cased), not what the user typed, since the directory matches `Bob` for `bob` and binds either; the role tests naming the login (`memberUid`, which matches case-exactly) use that form too. In proxy mode the proxy must pass that form (`AuthLDAPRemoteUserAttribute` in the Apache example). - Rate limits per user (evaluations, starts; per client address in anonymous mode, where a new pseudonym is free) and per client address (logins); the size of the answers is capped; the reports reading thousands of rows (the Statistics view, the summary of a lab on the Archives page) run two at a time, a third waits ten seconds for a slot then gets a 503 to retry. The sign-in page shows its own messages only (`/login?error=`: `password`, `rate_limited`, `directory`), never a text taken from the URL. - The `answers/` directory of a project is world-writable (the Qt GUI of the student writes there) and the agent, as root, merges the cheat answers `sre state` leaves in it (`answers/cheat.json`): that file is opened without following a symbolic link and read only when it is a regular file with one link, owned by root, `sre` or the reader, of at most `cheat_file_max_size` bytes (one million); anything else planted under that name is removed (the entry, never what a link points to) and logged once. - The connection log (`[access_log]`) keeps who did what from where: the sessions with their duration and projects, the logins and their failures, the denied calls, the administration calls. The only header it reads is `X-Forwarded-For`, from the peers of `trusted_proxies`, for the client address; the identity never comes from it. With `[access_log.database]` the same lines are in a MariaDB / MySQL table, read on the History page by the administrators only (`[auth] administrators` and the administrator group tests: instructors, never students; every consultation is itself logged); the account of the portal needs `SELECT, INSERT, DELETE` on that table and nothing else. - A project's owner is the third field of its running lab name; the portal, the agent and, in user mode, the `sre` CLI itself (`params.check_project_owner`: the local GUI of a pool host lists and reaches the same projects as the CLI lets through) check it. An instructor (see *Instructors* above) reaches every project, but the agent never runs a command as the student for them: a non-owner's `eval`, `state` and `connect` are privileged `sre` calls under the instructor's own login, the answers a non-owner writes go into the student's file with the student's metadata kept (the file never names the instructor), and a private terminal is listed to instructors only; a terminal an instructor opens on a device the student may not open themself (a closed machine or switch, a hidden machine) is private whatever the page asks, since a shared session is joined without the device check. The operations log of an instructor-mode project goes to the subscribers allowed to read it only. - What the applications of a project's desktop copy (the X selection, which any X client of the containers can set) is received by the owner's browser only, where the Display tab puts it into the clipboard; the page of an instructor helping on the project does not listen to it. Likewise the hyperlinks a program prints in a terminal (OSC 8 sequences) open, after a confirmation naming the target, on the owner's page only; on an instructor's or a spectator's page they are inert, the target shown as a tooltip. - The spectators of a presentation reach the presented project through the portal only, on a short list of read-only calls (the project, its events, the schema, the log, a terminal to *join*, the display; the evaluation table comes from the presenter's page through the portal, never from the agent), relayed to the agent under the presenter's identity (the agent trusts the portal for every identity anyway); everything else is refused, the portal never forwards what a spectator's browser sends on a terminal socket and the agent attaches a read-only client that cannot type nor resize, and the display socket of a spectator goes through a filter dropping the RFB input and resize messages (an unexpected message closes it). From the browsers, the presentation socket takes the pointer frames of the presenter's page only (a JSON object of at most 1 KiB locating the mouse, at most 60 a second, every other frame ignored; a spectator's frames are ignored); the pointer and the selection name nodes of the page and cells of a terminal, never a pixel, and a spectator's page resolves them on its own layout. The presentations live in the portal's memory. **The virtual desktops.** Each `Xvnc` listens on TCP port 6000+N on every interface of the pool host, protected by the MIT cookie of its project only, exactly like the X server of a workstation configured for the labs (`installation.md`); an nftables rule such as `nft add rule inet filter input tcp dport 6010-6099 iifname != "docker0" drop` keeps the other hosts out. The RFB socket of a desktop is reachable by root only (the agent); the window manager runs as `sreweb-x11` with no menu, no launcher and no key binding running a program of the host; the browser reaches a desktop through the owner check of the portal and of the agent. ## Limitations - After an update of the tree, the portal and the agents must be restarted for the Python code (`sre-web-tool restart --all` for the agents, see *Restarting the agents* above; the templates are re-read when they change); the scripts and style sheets of `web/static` are served with `Cache-Control: no-cache`, so a browser revalidates them on every page load (an `ETag` answers 304 for an unchanged file) and never runs a script older than the page. A page loaded before the update keeps its scripts until it is reloaded. - The virtual X desktops die with the agent (a restart of its unit): they are started again, empty, at the next use of the Display tab; the applications shown must be launched again. - The preloads of the Docker images are counted per portal (two portals sharing hosts each give `max_preloads` turns), need the portal (an agent never preloads by itself), are asked at the start of an agent or by hand only (no periodic check: a cron job running `sre-web-tool preload --all` does it) and are not kept across a restart of an agent: a preload asked by hand and not started yet is forgotten, one in progress is stopped (`[preload] at_start` asks again). A host that cannot be polled while it preloads stops counting, so one more host may get a turn meanwhile. - The `ports` of the bridged machines are bound on the pool host and shown with its name. - The exam mode of the Qt GUI (`exam.json` on a host) still excludes that host from the pool: the web site has its own exams (*Exams* above), per student rather than per host. The save of the terminal sessions across a restart of the agent is a later phase. - A terminal is lost when the agent restarts; its sub-tab offers a Reconnect button. A page that left keeps its terminals for `terminals_linger` only (120 minutes by default); where each one was (a sub-tab, a window) is remembered by the browser, another browser of the same user gets them back as sub-tabs. - The replay a joining viewer gets may start inside an escape sequence; the next prompt or redraw of the application settles the display. - The `[limits] eval_interval` of the portal (10 s between two evaluations of a login) also applies to an instructor evaluating several students' projects in a row. - A presentation ends when the portal restarts (nothing is kept on disk); the agents log the spectators under the presenter's login (the connection log of the portal has the real ones); a spectator's terminal shows the presenter's grid at a smaller font when their window is smaller; the presenter's pointer and selection are shown where the student's page holds the same element: a student reading the lab in another language sees the pointer on the matching paragraph and no selection there; in a terminal they follow the text (the rows are counted from the cursor, a selection is found again by the text of its first and last rows), so a selection of rows older than the student's 64 KiB replay is not shown, the student's terminal does not follow the scrollback position of the presenter's (a pointer on rows the student's screen does not show is hidden), and when several users type in the presented terminal its pty takes the smallest window, so a wider presenter's terminal wraps the long lines elsewhere and the pointer and the selection on them may miss; the pointer is not shown while the presenter's page has no socket open; a presented project never idles out while it is watched. A focus ends when the portal restarts too; while one is on, the student's other browser tabs all go to the project. - A web conference is forgotten when the portal restarts (the meeting goes on on the BigBlueButton server until it expires); every page of the site joins the meeting again (a few seconds); two tabs of one user hand the client over to each other, but two pages loading at the same instant may both join. ## Development `make web-tests` runs the offline tests of the web front-end: the agent against a fake `sre` (`tests/mock_sre.py`), the portal against that agent, the templates. `python3 -m sreweb.agent --config agent.toml --listen 127.0.0.1:8451` and `python3 -m sreweb.portal --config web.toml --listen 127.0.0.1:8450 --web-dir web` (from `src/`, in the venv) run both locally; with `sre_exe` pointed at a wrapper of `tests/mock_sre.py` (`SRE_PUB_DIR` exported in the wrapper), the whole site works without Docker. The French catalog is `locale/fr/LC_MESSAGES/sreweb.po` (hand-maintained, `make web-translations`). The pure JavaScript modules (the lab tree of the Open page, the group keys, the exam form, the floating terminal windows) are tested under `node` or `bun`; on a machine with neither, `make tests` runs them under bun in a throwaway container of the `oven/bun` image instead (`SRE_TEST_JS=docker`, what the Makefile passes then: the image is pulled once, the tests skip without a reachable Docker daemon, root or the `docker` group; `SRE_TEST_JS=docker:` names another image, `SRE_TEST_JS=` leaves them skipped). Installing a runtime (`apt install nodejs`) is the lighter alternative. ## Stress test `scripts/web-stress.py` simulates a class of students on a portal through its HTTP / WebSocket API, as the browser drives it: for each student (logins `stress001`, `stress002`...), the Projects page (`GET /api/projects` every 5 s, the presence socket), the Open page (`GET /api/labs`), `POST /api/projects` read as NDJSON until the `result` line (a 503 or 429 is retried after its `retry_after`), the project page (the project JSON, the schema, the events socket kept open and reconnected with the browser's backoff, `POST .../touch` every 60 s), a terminal on the first device open to the student with a few commands typed (the echo of the first one is checked), then, over `--hold` seconds, `--answers` saves and `--evals` self-evaluations, and finally the stop. The students begin spread over `--ramp` seconds. `--instructor` adds an instructor on the Watch page (`GET /api/watch`, the Pool page's calls, every 5 s), `--admin-token` samples `GET /api/pool` every 5 s (the `reachable` flag and the project count of each host as the portal sees them), `--probe` measures `GET /api/me` every 2 s. The tool needs Python 3.11 and aiohttp only (the venv of the installation has them) and runs in **proxy authentication mode**: the identity of each student travels in the `X-Remote-*` headers, so it must run on a host of `trusted_proxies` (the portal's own host, against `http://127.0.0.1:8450`); the instructor is made by `X-Remote-Gid: 2000` (`--instructor-header` / `--instructor-value` otherwise). The summary on stdout gives, per phase (`start`, `project_get`, `events_ws`, `terminal_ws`, `eval`, `stop`...), the number of calls, the errors, the latencies (p50 / p95 / max) and the statuses, then the students completed, the checks failed (`own_projects_only`: a listing showed another student's project; `project_listed`: the new project was missing from `GET /api/projects`; `terminal_echo`; `eval_grades`; `events_stayed_open`; `no_unexpected_removed`; `stopped`), the hosts used and the pool samples (the flips of `reachable` show the status polls timing out). `--report FILE` writes everything as JSON (one entry per student), `--events FILE` a CSV timeline (one line per request or socket event). The exit status is 0 when every student completed with every check true. `--keep` leaves the projects running; `--cleanup` stops every project of the synthetic logins (through the instructor identity) and exits. ### Stage 1: a fake pool, no Docker `scripts/web-fake-pool.py` launches from the checkout N real agents whose `sre` is a wrapper of `tests/mock_sre.py` (the projects are directories, the terminals `/bin/sh`, a start takes 7 × `--start-delay` seconds, an evaluation `--eval-delay`), and a real portal in proxy mode trusting the loopback, with the production `[limits]` (`--start-interval` / `--eval-interval` override them), `max_projects` per host and an admin token; then it runs the command given after `--` with `SRE_STRESS_PORTAL`, `SRE_STRESS_ADMIN_TOKEN` and `SRE_STRESS_WORKDIR` in its environment (the defaults of `--portal` and `--admin-token`) and stops everything. Every `MOCK_*` variable of its environment reaches the fake `sre` (`MOCK_NO_MARK=1`: the letters of a self-evaluation, `MOCK_EVAL_ERRORS`, `MOCK_COOLDOWN`, see the header of `tests/mock_sre.py`): ```sh venv/bin/python3 scripts/web-fake-pool.py --hosts 4 --max-projects 25 -- \ venv/bin/python3 scripts/web-stress.py --users 100 --ramp 60 --hold 120 --instructor --probe \ --report report.json --events events.csv ``` Without a command the pool stays up until Ctrl-C (`portal=... admin_token=... workdir=...` is printed first), for runs by hand. The work directory (configs, `portal.log`, `access.log`, one `agent.log` and `sre.log` per host) is removed at the end unless `--keep-workdir` or a daemon died; the end of the log of a daemon that died is printed with the failure. The agents listen on the ports after the portal's (`--listen`, 18450 by default): keep them below the ephemeral ports of the system (32768 on Linux), where a port may be held by a client connection, even for a minute after it closed. `--administrators prof,...` makes those logins administrators of the portal (`[auth] administrators`), so that an instructor among them (`X-Remote-Gid: 2000`) can exercise the History page; its Sessions view lists the open sessions without any database. ### Stage 2: the production pool On the portal's host, outside class hours, with the tool copied there (or `/opt/sre/scripts/` once synced) and the venv of the installation: 1. **Preflight.** The portal must see the tool as a trusted peer: from the host itself when the portal container uses the host network (`docker inspect sre-web --format '{{.HostConfig.NetworkMode}}'`), from inside the container otherwise (`docker exec`). Read the admin token in `[admin]` of `web.toml`. `sre-web-tool status` and `list`: the hosts up and the spare slots; with fewer slots than students, the pool wakes hosts during the ramp (`wake_timeout`, the students refused with 503 "a host is starting" retry after the `retry_after` of the answer, 30 s). Check `ulimit -n` of the portal process (two relayed sockets per student). Note the `archive_dirs` of the hosts: every evaluation writes an archive there. 2. **Smoke run:** `web-stress.py --portal http://127.0.0.1:8450 --users 3 --hold 60 --lab sre/dhcp.py --instructor --admin-token ... --probe` must give 3 students completed and no check failed. 3. **Full run:** the same with `--users 100 --ramp 300 --hold 600 --report ... --events ...`. Meanwhile: `docker stats sre-web` (the CPU of one event loop, the memory), the connection log (`event=start ... result=`, `denied`, hosts unreachable), `sre-web-tool list` (states, projects per host), the journal of an agent (`journalctl -u sre-agent -f`), the probe's p95 in the summary. 4. **Cleanup:** `web-stress.py --cleanup ...` stops what a stopped run left (a Ctrl-C stops the students' projects by itself), then remove the archives of the synthetic logins (`*@@@stress0*` under `archive_dirs` on the hosts) and check `sre-web-tool list` is back to 0 projects. The reverse proxy and the directory are bypassed (the tool speaks to the portal), and `follow_local_sessions` never applies (the synthetic logins have no local session).