Web server (remote use)

The web site lets the students work on the labs from a browser, without the Qt interface and without a remote desktop: the projects run on a pool of Docker hosts, every tab of the graphical interface (Schema, Informations, Machines, Networks, Questions, Evaluation, Terminals, Apply Configuration, Log) is rendered in the browser and the terminals go through WebSockets. The users log in with an LDAP directory (the portal binds to it, or an Apache reverse proxy in front authenticates them), or everybody is anonymous.

Nothing of a lab ever reaches the browser but what a student sees today: the content of info.json (resolved to the user’s language, the instructor texts stripped), the outputs of the sre --user commands (the schema, the grades of a self-evaluation, the console of a machine). The lab files, data.json, the hidden machines and the archives stay on the pool hosts.

The instructors get, on top of what a student sees (Instructors below):

• a Watch page, the web counterpart of sre watch: every running project of the pool, one row per project with its owner, its lab, the last time the student did something on it, the grade and mark of its last evaluation, its age and its errors, the number of self-evaluations the student has run, refreshed every few seconds;

• an Archives page: every evaluation of the pool hosts, the closed projects included, kept in a database (the results, never the answers), searched by student, group, lab, host and period, with the detail of an evaluation and the summary matrix of a lab;

• the projects of the students, open to them to help: every tab of the project, and the student’s terminals joined when the Terminals tab opens, so that the instructor and the student see the same shell and both type in it (a private terminal, unseen by the student, is offered too);

• presentations: a project, the instructor’s own or a student’s, duplicated read-only on the browsers of the whole class or of chosen students and groups to demonstrate something. What the instructor does on their own page (changing tabs, typing in a terminal, selecting a question, changing the drawing of the schema, …) is shown at once on the students’ browsers;

• the focus on one student: while helping a student on their project, the instructor brings the student’s browser to that project and keeps the two pages on the same tab, the same terminal and the same question, whichever of the two changes it;

• the web conference: an instructor opens a BigBlueButton room for chosen students or groups (or the whole class) and talks to them, camera and screen if they wish, in a window over the pages of the site while everybody goes on working (Web conference below).

Architecture

browser ──HTTPS──► sre-web (the portal) ──HTTP + token──► sre-agent (one per pool host) ──► sre --user …

• sre-web, the portal, is the web site the browsers talk to. It authenticates the users, keeps a signed session cookie, knows the pool of hosts and relays every request about a project to the agent of the host running it. It runs as an unprivileged user, needs neither Docker, Kathara nor the labs, and may run anywhere on the network (the Debian package sysreseval-web installs alone).

• sre-agent runs on every pool host, as root, next to sre. It drives the sre CLI in user mode on behalf of the web users exactly as sre-wrapper + sudo do for the students of a workstation (sre --user <cmd> with SUDO_USER=<login>), owns the terminals (a pseudo-terminal per sre connect), writes the answers file with the user’s identity (atomically and under the advisory lock on the answers/ directory the Qt GUI takes too, the owner of a file the Qt GUI of the same student created kept, so that the GUI goes on writing it; what that GUI writes is pushed to the open pages, which show the same answers), merges the cheat answers of the states, tails the operations log and schedules what the Qt GUI used to schedule: the periodic evaluations of a lab (eval_interval_without_exam_mode), the evaluation before a project is closed (eval_before_exit), and an idle reaper closing the projects nobody touched for the configured time. For an exam project of the web site (Exams below) it applies the rules of the Qt GUI’s exam mode to that project alone: the evaluations at the period of the exam, the final one at its end, the owner cut off then.

• sre-web-tool is the command line next to the portal: sre-web-tool … lists and changes the pool of the running portal through its administration API (The pool below).

• The pool hosts share the lab tree as today (NFS, rsync or the Debian package) and, if the instructor wants sre watch / sre sheet / sre outline across the pool, an archive_dirs entry on a shared directory. The hostname of the archives is the pool host. The Watch page of the instructors needs none of that: each agent reads the archives of its own host (params.archive_dirs) on demand.

• Placement: a new project goes to the host where its user has a local login session (the workstation they sit at, follow_local_sessions), else to the reachable host with the lowest projects / max_projects (then the lowest load); a host in exam mode is never used, a host with instructors_only = true never for a student (an instructor may also name the host on the Open page); the whole rule is under Placement in A pool of workstations. A classroom workstation holds about thirty projects (210 containers in 3.4 GB).

• The lab list of the Open page comes from one host, the one a new project of the user would go to (every host of a pool serves the same lab tree; the next candidate is asked when it fails), and is kept labs_cache_ttl seconds (60) per role, since the reserved hosts are the instructors’ only: opening the page costs one sre list per minute whatever the size of the pool.

The agent enforces what the sre CLI does not check: the exact running lab name (the CLI resolves substrings) and its owner (a user only reaches their own projects, an instructor every project; the CLI’s own check, params.check_project_owner, lets a --user caller reach their own projects and root’s only, which is why an instructor is served by privileged calls; the wipe, stop-all, exec and host-wide exam commands have no endpoint: an exam of the web site is a POST /v1/projects with an exam object, adjusted with PUT / DELETE …/exam), allow_connection of the devices, allow_self_grade, the user-allowed states, the export_schema_pdf flag of the lab (the Export Schema to PDF button). Save, restore and the Kathara export are not offered on the web. An instructor acting on somebody else’s project is served by a privileged sre (no --user, their own login: sre eval without cooldown nor counter, sre connect to every listed device, sre state), never by a command run as the student; the owner, instructor or not, follows the student path.

Installation

From a checkout

make venv installs the Python dependencies (aiohttp, jinja2, aiohttp-jinja2, ldap3, and pymysql for the optional database of the connection log); make xterm and make novnc (run by make install) download the terminal library and the VNC viewer of the browser into web/static/vendor/ (checksum-pinned, not committed). scripts/install.sh offers to install the systemd units sre-agent.service and sre-web.service (disabled) and the example configuration files agent.toml and web.toml in /etc/sreweb/ (params.agent_config_file / params.web_config_file; --config names another file).

On every pool host: write /etc/sreweb/agent.toml (root, mode 0600) with a long random token (the example file listens on every interface, 0.0.0.0:8451: allowed_clients and the mutual TLS below keep the port to the portal), then systemctl enable --now sre-agent. The unit starts the agent only when that file exists and /etc/sreweb/DO_NOT_START_AGENT does not (ConditionPathExists=: one image for a whole classroom may ship the unit enabled, the hosts without a configuration file, or with the marker, skip it without any failure; checked at boot and on systemctl start / restart), and it does not restart the agent after a configuration error (exit status 78, EX_CONFIG: a bad agent.toml, a missing certificate; the message is in the journal, fix the file then systemctl restart sre-agent); five failed starts within five minutes leave it stopped until the next boot or systemctl reset-failed. For the Display tab (below), install tigervnc-standalone-server and openbox, create the sreweb-x11 system user (scripts/install.sh offers it) and set [display] enabled = true. For a whole classroom, scripts/make-web-hosts.sh (see A whole site at once below) writes the tokens, the certificates and the [[hosts]] entries from a list of machines.

On the portal host: create the sre-web system user (adduser --system --group --no-create-home --shell /usr/sbin/nologin sre-web), write /etc/sreweb/web.toml (root:sre-web, mode 0640: it holds the cookie secret and the agent tokens; the files it imports likewise), then systemctl enable --now sre-web. Put a reverse proxy doing TLS in front (scripts/etc/sre-web-nginx.conf is an nginx example, sre-web-apache.conf an Apache one that also authenticates the users: the proxy mode below), or give the portal its own certificate (tls_cert / tls_key).

Debian packages

sysreseval-agent on every pool host (it depends on sysreseval and sysreseval-web, whose code it shares), sysreseval-web on the portal host (it depends on nothing of SysResEval). The units are sysreseval-agent.service and sysreseval-web.service, disabled by default (the agent’s with the conditions and restart rules above, on /etc/sysreseval/agent.toml and /etc/sysreseval/DO_NOT_START_AGENT); the configuration files are /etc/sysreseval/agent.toml and /etc/sysreseval/web.toml (examples in /usr/share/doc/sysreseval-*/examples/); the postinst of sysreseval-web creates the sre-web user and protects web.toml and web.d/*.toml when they exist.

The postinst of sysreseval-agent creates the sreweb-x11 user of the virtual X desktops; tigervnc-standalone-server and openbox are its Recommends (the Display tab is off until [display] enabled = true).

Configuration

agent.toml

Key	Default	Meaning
listen	127.0.0.1:8451	the address the portal reaches: 0.0.0.0:8451 in the example file (every interface, the Docker bridge the students’ containers reach included: set allowed_clients and tls_ca), or a management interface the students do not reach
token	required	the shared secret of this host, repeated in the portal’s [[hosts]] entry (32+ characters)
sre_exe	the installation’s sre	the command run for the users
poll_interval	1	seconds between two scans of the projects directory (the Qt GUI polls every second too)
max_idle	24h	default idle timeout of a project (the portal gives its own per user); 0 = never
terminals_per_project, terminals_per_device	16, 4	caps on the open terminals (every Connect opens a new terminal on the machine); at a cap, the idle terminal that has waited longest gives way
terminals_linger	120m	how long a terminal whose last browser left (the page closed or reloaded, another project opened, the network cut) is kept for the browser to come back to it; 0 = it ends at once; the × of a terminal ends it at once in any case
max_concurrent_starts, max_concurrent_evals	2, 2	concurrent sre start / sre eval on the host
lab_list_cache_ttl	60	seconds the output of sre list is cached
state_dir	/var/lib/sre/web-agent	the idle clocks of the projects, kept across restarts
tls_cert, tls_key	none	the certificate and key of this agent (TLS to the portal)
tls_ca	none	the CA a client certificate must be signed by: only the portal gets past the handshake (mutual TLS)
allowed_clients	[] (any)	the addresses or networks the agent answers, e.g. ["10.0.0.1", "10.0.1.0/24"]
occupied_check	both	when the host reports itself occupied (no new project, never powered off): sessions (a local login session: loginctl, or who), projects (projects of the Qt GUI on the host), both, never
close_projects_on_stop	shutdown	what the agent does with the projects it manages when its unit stops: shutdown closes them (last evaluation, sre stop) only when the system is going down, always, never; within shutdown_timeout (240 s, the unit’s TimeoutStopSec is 300)
allow_power_off	false	the portal may power the host off (POST /v1/power/off, refused when occupied or with any project) with power_off_command (systemctl poweroff)
restart_command	none	how the agent restarts itself on sre-web-tool restart (POST /v1/restart): by default systemctl restart of its own unit, read from /proc/self/cgroup; a command for an agent run otherwise (refused, 409, when neither applies)
[timeouts]	start = 900, eval = 600, state = 600, stop = 300, list = 30, report = 600	seconds, per sre command (report: the spreadsheet / PDF reports of the Archives page, built by sre report on this host; export, the timeout of the former Kathara export, is still accepted)
[display]	enabled = false	one virtual X desktop per project whose lab has an x11_host machine, shown in the Display tab (see The Display tab below): user (sreweb-x11, the unprivileged account the X processes run as), xvnc_command (Xvnc, TigerVNC), window_manager_command (openbox, "" = none), geometry (1280x800), depth (24), frame_rate (30), first_display / last_display (10 / 99: display :N listens on TCP port 6000+N for the containers), runtime_dir (/run/sre-agent: the RFB sockets and X authority files), start_timeout (10 s for the X server to come up)
[preload]	at_start = false	the Docker images of the labs pulled by the agent (sre preload-images) when the portal gives the host its turn (see Pre-loading the images below): at_start (the agent asks for a preload each time it starts; sre-web-tool preload asks by hand in any case), paths (the lab files and directories scanned; default: the lab directories of the installation), timeout ("2h": a preload still running after this long is stopped)

Durations are written as 90, "90s", "30m", "2h" or "1d". An unknown key is refused. import (a path or a list of paths relative to agent.toml: a directory, a glob or a file) merges other files into this one, as for web.toml below.

web.toml

Key	Default	Meaning
listen	127.0.0.1:8450	behind a reverse proxy: the loopback
public_url	none	the URL the users type (the Origin of the WebSockets is checked against it)
secret	required	signs the session cookies (32+ random characters)
trusted_proxies	[]	proxy mode: the addresses or networks of the reverse proxy, the only peers whose identity headers are trusted (the TCP peer address, never X-Forwarded-For); required in that mode. In every mode, the connection log takes the client address from the X-Forwarded-For header of these peers
languages	["en", "fr"]	the languages offered at login, the first one by default
tls_cert, tls_key	none	own TLS instead of a proxy
[auth] mode	anonymous	anonymous: a pseudonym guest-xxxxxxxx per browser (the user may type a display name); ldap: the directory account; proxy: the identity the reverse proxy passes in headers (below)
[auth] administrators	[]	the logins of the administrators: instructors who may read the connection log on the History page (Administrators below); a login that is not an instructor gets nothing. By group: administrator_groups / administrator_group_filter of [auth.ldap], administrator_groups of [auth.proxy]
[auth.ldap]		uri, then either bind_dn_template = "uid={login},ou=people,…" (bind as the user) or search_base + search_filter (+ bind_dn / bind_password of a service account) to find the user first; attr_fullname (cn), attr_mail (mail), attr_group (none; the attribute holding the student group, e.g. departmentNumber, its first value: the Group column and filter of the Watch page); the instructor tests (below): instructor_groups (DNs; memberOf, or group_filter = "(member={dn})"), instructor_filter ("(gidNumber=2000)", "" = none), group_search_base (none = no membership test) + instructor_group_filter ("(&(objectClass=posixGroup)(gidNumber=2000)(memberUid={login}))"); the administrator tests (Administrators below): administrator_groups (DNs, tested like instructor_groups) and administrator_group_filter (a group search under group_search_base, none by default); the connection: ldaps:// or ldap:// + start_tls, the server certificate always verified against the system CA store or tls_ca (the absolute path of a PEM file: a CA bundle, or the self-signed certificate of the server itself) and the host of the uri against the names of the certificate (subjectAltName, else the CN: name the server as its certificate does, not by address); a plain ldap:// without start_tls is refused at startup unless allow_cleartext = true (the passwords in LDAP mode, the roles, groups and administrators in every mode would travel unprotected); timeout. The login attribute: attr_login (none = the attribute of a search_filter of the form (attribute={login}), else the RDN attribute of bind_dn_template); in LDAP mode the identity carries its stored value, whatever case the user typed (the directory matches Bob for bob: one user, not two), and the Groups page (User groups below) lists the people by it, in every mode, with groups_filter ("(objectClass=posixGroup)": the groups listed under group_search_base; "" = none), attr_group_name (cn), attr_group_members (memberUid; a value may be the login, the uid or the DN of a person). Also read in proxy mode with ldap_roles / ldap_groups
[auth.proxy]		header_login (X-Remote-User), header_fullname (X-Remote-Name), header_email (X-Remote-Email), header_groups (X-Remote-Groups): the headers set by the proxy; groups_separator (;); header_group (none; the header holding the student group, e.g. X-Remote-Group from departmentNumber, its first value); instructor_groups: the values of the groups header giving the instructor role; instructor_header + instructor_values (e.g. X-Remote-Gid and ["2000"]): a second header test; administrator_groups: the values of the groups header making an instructor an administrator; ldap_roles (false): the portal runs the tests of [auth.ldap] itself for the login the proxy gives, and reads its attr_group, the result cached per login for ldap_roles_ttl (10m); ldap_groups (false): the group part alone (attr_group of [auth.ldap] for a user whose group header is empty, and for the owners of the Watch page the agents know no group for; the roles stay those of the headers; ldap_roles implies it); logout_url: where Logout sends the browser (the /apache-logout location of the Apache example; none: no Logout button)
[pool]		idle_timeout_anonymous (2h), idle_timeout_authenticated (24h): the idle timeout given to the agents per user, 0 = never; poll_interval (2): seconds between two polls of the agents; ca: the CA the agents’ certificates are verified against (the system store without it); client_cert, client_key: the client certificate of the portal, presented to every agent; the elastic pool (below): spare_projects (5), min_hosts_up (1), active_hours (07:00-20:00, local time, "" = always), wake_timeout (180), wake_interval (30), wol_broadcast (255.255.255.255), idle_host_timeout (0 = never); follow_local_sessions (true): a user logged in at the keyboard of a pool host gets their new projects there whatever its load, below its max_projects (Occupied hosts below); labs_cache_ttl (60): seconds the lab list of the Open page, asked from one host, is kept per role (0 = asked at every open); max_preloads (2): the hosts pulling the Docker images of the labs at the same time (Pre-loading the images below; 0 = the portal lets none)
[limits]		eval_interval (10 s between two self-evaluations of a user), start_interval (30 s between two starts of a user; a start that created no project — the flavor form shown or refused, no host available, the agent not answering — gives the slot back at once), login_per_minute (5 per client address), max_answer_bytes; in anonymous mode eval_interval and start_interval count per client address too, since a new pseudonym costs one POST /login
[admin]		token: the token of sre-web-tool … (32+ characters; none = the command is refused), accepted from the loopback only unless listen_any = true; state_file: what is changed by hand in the pool (the hosts added, removed, enabled, disabled, drained), kept across the restarts of the portal (/var/lib/sre-web/pool.json when that directory exists, else pool.json next to web.toml; Restarting the portal below)
[access_log]		the connection log (below): file (none; the absolute path of the log file, e.g. /var/log/sre-web/access.log, rotated daily), keep (400 days of rotated files kept; 0 = no rotation, for logrotate), journal (true: the lines also go to the portal’s own log: stderr, the journal, docker logs), session_timeout (30m of inactivity closes a session); database (the next row)
[access_log.database]	none	the same events in a MariaDB / MySQL table (below): host or unix_socket, port (3306), user, password (in an imported secrets file), name (the database), table (access_log, created when missing), keep (400 days of rows kept, purged daily; 0 = forever), queue (10000 rows kept in memory while the server does not answer), connect_timeout (5); needs PyMySQL (python3-pymysql, a Suggests of the Debian package: optional)
[archives]		the archives of the evaluations of the pool hosts in a second table of that database (The archives of the evaluations below; on iff [access_log.database] is set): table (archives, created when missing), keep (0 = the rows are kept forever; else days of rows kept, purged daily), poll_interval (30s between two collections from the agents)
[groups]		the user groups of the Groups page (User groups below): table (user_groups: the custom groups the instructors create, in the database of [access_log.database], their members in <table>_members; both created when missing), directory_ttl (10m: how long the people and groups read from the directory are kept), refresh_interval (10s: the least time between two reads forced by the Refresh button of the page; the clicks arriving during a read share it, 0 = none)
[exams]		the exams of the Exams page (Exams below; on iff [access_log.database] is set): the defaults of a new exam, each adjustable per exam — lead (10m: how long before the start the other projects of the students are hidden and the countdown shown), pre_start (5m: how long before the start their exam projects are started in the background), eval_interval (60s: the period of the evaluations the agents run during the exam) —, close_after (1h: how long after the end of a student’s exam their exam projects are closed; 0 = never, an instructor closes them), table (exams, with <table>_members, both created when missing), keep (0 = the past exams are kept forever; else days, purged daily)
[site]	none	the institution shown next to the SRE logo in the header of every page, the login page included: logo (the absolute path of an image file, JPG / PNG / SVG…, readable by sre-web, served at /site/logo), name (a string, or one per language: { fr = "IUT de Démo", en = "Demo Institute" }, the first configured language when the page’s one is missing), url (the page the two link to; none: plain text)
[bbb]	none	the BigBlueButton server behind the Webconf button of the instructors (Web conference below): url (the server, or its bigbluebutton/ / bigbluebutton/api/ path) and secret (the shared secret of its API, bbb-conf --secret; both or none), checksum (sha1; sha256 or sha512 when the server is restricted to them), welcome (the welcome message of the client: a string, or one per language; the server’s message by default), record (false), mute_on_start (false), duration (0 = unlimited; a duration, "2h": the server ends the meeting after it), max_participants (0 = unlimited), poll_interval (10s: how often the participants and the ended meetings are read from the server), [bbb.userdata] (the bbb_* client settings sent with every join, e.g. bbb_skip_check_audio = true), private_rooms (true: the Talk privately button of an instructor on a student’s project, Private conversations below; false removes it), [bbb.private_userdata] (the bbb_* client settings of the joins of a private conversation, over [bbb.userdata] and the portal’s own)
[[hosts]]	required	one per pool host: name (what the URLs show), url of its agent, token of its agent.toml, max_projects (25); to wake it, mac (Wake-on-LAN) or wake_command ({name}, {mac} substituted); power = true lets the portal power it off; enabled = false keeps it polled without placing projects; instructors_only = true reserves it for the instructors’ projects (a student is never placed there and does not see its labs; a project already running there stays reachable; outside the elastic pool, see Reserved hosts)
import	none	other files merged into this one before it is checked: a path or a list of paths, relative to the directory of web.toml: a directory (its *.toml files, by name), a glob pattern ("web.d/*.toml", possibly matching nothing) or a file. Tables merge, [[hosts]] entries add up; a key given in two files is refused, naming both; an imported file cannot import

import = ["web.d"] splits the configuration into /etc/sysreseval/web.toml and the *.toml files of /etc/sysreseval/web.d/, read by name after it: a file per pool host, the secrets apart. Nothing is silently overridden: a key given twice is an error naming both files, and an error found after the merge names web.toml. The imported files hold the same secrets as web.toml and need the same owner and mode (root:sre-web, 0640).

The full name and the e-mail of an LDAP account (attr_fullname, attr_mail) go into the answers file of the projects like the GECOS fields do on a workstation, so sre outline and sre sheet show them; an anonymous user is identified by the pseudonym and the display name. The student group (attr_group in LDAP mode, header_group or ldap_roles / ldap_groups + attr_group in proxy mode; departmentNumber typically, its first value, capped at 64 characters) goes there too under the key group (hence into the archives): the Watch page shows it and filters on it. With a directory the portal can read (attr_group in LDAP mode, ldap_roles or ldap_groups in proxy mode), the Watch page also looks up the group of the owners the agents know none for, first of all the projects started on the Qt GUI, whose answers file never carries one, and shows for the projects of the Qt GUI the full name of the directory (attr_fullname) rather than the GECOS name of the account their answers file carries (lower case on some sites): one cached lookup per login. The agent writes that metadata as soon as a project is created, before any answer, and again when the owner opens the project page and the file lacks it or holds other values (so a project created by an older agent, or on the Qt GUI, gets its name and group then); a request whose identity carries no full name or no group (a header the proxy did not set) never erases the stored values.

Authentication by the reverse proxy

With [auth] mode = "proxy" the portal has no login page: the reverse proxy in front authenticates the users itself and names them in request headers, which the portal reads on every request (pages, API calls, the NDJSON streams of the starts, the WebSockets of the terminals). Nothing is stored in a cookie: the proxy’s session is the session, and a request that does not come from a peer of trusted_proxies has no identity whatever its headers. scripts/etc/sre-web-apache.conf is the Apache 2.4 example (mod_authnz_ldap + mod_proxy):

<Location "/">
    AuthType Basic
    AuthName "SRE"
    AuthBasicProvider ldap
    AuthLDAPURL "ldaps://ldap.example.org/ou=people,dc=example,dc=org?uid,cn,mail,memberOf,gidNumber,departmentNumber?sub?(objectClass=inetOrgPerson)"
    AuthLDAPRemoteUserAttribute uid
    Require valid-user
    RequestHeader set X-Remote-User "expr=%{REMOTE_USER}"
    RequestHeader unset X-Remote-Name
    RequestHeader set X-Remote-Name "%{AUTHENTICATE_CN}e" env=AUTHENTICATE_CN
    RequestHeader unset X-Remote-Email
    RequestHeader set X-Remote-Email "%{AUTHENTICATE_MAIL}e" env=AUTHENTICATE_MAIL
    RequestHeader unset X-Remote-Groups
    RequestHeader set X-Remote-Groups "%{AUTHENTICATE_MEMBEROF}e" env=AUTHENTICATE_MEMBEROF
    RequestHeader unset X-Remote-Gid
    RequestHeader set X-Remote-Gid "%{AUTHENTICATE_GIDNUMBER}e" env=AUTHENTICATE_GIDNUMBER
    RequestHeader unset X-Remote-Group
    RequestHeader set X-Remote-Group "%{AUTHENTICATE_DEPARTMENTNUMBER}e" env=AUTHENTICATE_DEPARTMENTNUMBER
    RequestHeader unset Authorization
</Location>
# Logout: a location that knows no user, so the browser forgets the credentials it cached
ProxyPass "/apache-logout" !
<Location "/apache-logout">
    AuthBasicProvider file
    AuthUserFile /dev/null
    Require valid-user
    ErrorDocument 401 "<!DOCTYPE html>... You are logged out. <a href='/'>Log in again</a> ..."
</Location>
ProxyPreserveHost On
ProxyPass        "/" "http://127.0.0.1:8450/" upgrade=websocket timeout=3600
ProxyPassReverse "/" "http://127.0.0.1:8450/"

AuthLDAPRemoteUserAttribute is not optional: it makes REMOTE_USER, hence the login header, the value the directory stores in the login attribute. Without it Apache passes what the user typed, and since the directory matches Bob for bob (uid is caseIgnoreMatch) one account would become two users of the portal (separate projects, out of the exam audiences and custom groups naming bob, split Watch and Archives rows). The attribute must be the one the agents and the exam audiences know the users by (uid for the Unix login of the workstations, see A pool of workstations). In LDAP mode the portal does this itself: the identity carries the stored value of the login attribute (attr_login, else the attribute of the search_filter or of the bind_dn_template), whatever case the user typed.

The rules that keep this safe:

• The portal listens on the loopback (or on an address only the proxy reaches) and trusted_proxies holds the proxy’s address as the portal sees it (127.0.0.1 and ::1 for a proxy on the same host); the portal takes the TCP peer address, never X-Forwarded-For (the connection log alone reads that header, from these peers only, for the client address).

• The proxy sets the four headers on every request, replacing whatever the browser sent (RequestHeader set does; never pass a client header through), after the authentication (not early). An optional header is removed when the attribute is missing, or mod_headers writes the literal (null), which the portal treats as empty.

• The attributes exported by mod_authnz_ldap (AUTHENTICATE_CN…) are those listed in the AuthLDAPURL; a multi-valued one is joined with ; (hence groups_separator = ";"). The groups header may carry any attribute, memberOf DNs or eduPersonAffiliation values: instructor_groups lists the values giving the instructor role; instructor_header / instructor_values test a second attribute the same way (gidNumber above, the primary group); header_group names the header carrying the student group (departmentNumber above). A group membership the proxy cannot export (a posixGroup listing its memberUid, without the memberof overlay) is tested by the portal itself with ldap_roles = true and the [auth.ldap] settings (see Who is an instructor). A long list of groups can exceed Apache’s LimitRequestFieldSize (8190 bytes): raise it or export fewer attributes.

• public_url must be the https:// URL the users type (the Secure flag of the language cookie, the Origin check of the WebSockets together with ProxyPreserveHost On).

• An HTTP Basic authentication has no session to end, but the browser forgets the credentials it cached as soon as the server refuses them: the /apache-logout location of the example knows no user (mod_authn_file on /dev/null, the realm inherited from the main location), so every request gets a 401 and the browser shows its password dialog once more; cancelling it displays the “logged out” page of the ErrorDocument (a password typed there is refused and the dialog comes back). logout_url = "/apache-logout" makes the Logout button appear; the button is hidden without it. A mod_auth_form login (sketched at the end of the example) has a real logout handler at the same URL.

• sre-web-tool … talks to the loopback port directly, never through the proxy (the Basic authentication would refuse its bearer token).

The connection log

[access_log] of web.toml makes the portal keep a log of its use: who connected from where and for how long, which projects were opened, started, evaluated, closed. One key=value line per event (a value with spaces or quotes is JSON-quoted, so a user agent or a display name cannot forge a line), in the file named by file (rotated at midnight, keep days of access.log.YYYY-MM-DD files kept; keep = 0 leaves the rotation to logrotate, the file is reopened when it moves) and, with journal = true (the default, and the whole log when no file is set), in the portal’s own log on stderr: the journal under systemd, docker logs in a container. The unit creates /var/log/sre-web (LogsDirectory=); a container mounts a directory for the file or writes it under /var/lib/sre-web. The lines of the file start with a timestamp (2026-10-26T09:12:03+0100):

event=login login=alice addr=10.0.1.7 mode=ldap role=student anon=0 name="Alice Test" group=INFO1
event=session_open login=alice addr=10.0.1.7 role=student anon=0 name="Alice Test" group=INFO1 ua="Mozilla/5.0 …"
event=start login=alice addr=10.0.1.7 lab=sre/dhcp.py instructor=0 debug=0 host=a01 placement=load result=created rln=1761466323@@@dhcp@@@alice status=200 duration=12.4
event=project_open login=alice addr=10.0.1.7 host=a01 rln=1761466323@@@dhcp@@@alice
event=terminal_open login=alice addr=10.0.1.7 host=a01 rln=1761466323@@@dhcp@@@alice device=m1 join=- private=0
event=terminal_close login=alice addr=10.0.1.7 host=a01 rln=1761466323@@@dhcp@@@alice device=m1 join=- private=0 duration=612.3 code=1000
event=eval login=alice addr=10.0.1.7 host=a01 rln=1761466323@@@dhcp@@@alice status=200 duration=8.1
event=project_close login=alice addr=10.0.1.7 host=a01 rln=1761466323@@@dhcp@@@alice duration=1803.6 code=1001
event=stop login=alice addr=10.0.1.7 host=a01 rln=1761466323@@@dhcp@@@alice status=200 duration=3.0
event=logout login=alice addr=10.0.1.7 status=200 duration=0.0
event=session_close login=alice addr=10.0.1.7 role=student name="Alice Test" group=INFO1 reason=logout since=2026-10-26T09:11:50+0100 duration=1850 requests=14 projects=1761466323@@@dhcp@@@alice

A session is the requests of one login from one client address (addr): it opens at the first request (session_open: role, pseudonym or not, name, student group, user agent) and closes (session_close) at a logout, after session_timeout without a request (reason=idle; the duration then ends at the last request, not at the expiry) or when the portal stops (shutdown); a session whose project page, terminal or desktop is still open never expires. The closing line repeats the role, the name and the group (the student group of the directory or of the proxy, attr_group / header_group, - without one; the last non-empty value seen during the session, since a directory lookup may fail on the first request) and gives the start, the duration in seconds, the number of requests (the polls of the pages and the keep-alive of a project are not counted) and the projects used: the Sessions view of the History page (Administrators below) is built from that line alone. In proxy mode there is no login: the first request of a user opens the session and the request following a logout opens another one. The other events:

• login (mode: anonymous or ldap) and login_failed (reason: password, rate_limited, directory when the directory does not answer, proxy_mode);

• start: the lab, the host chosen and placement, why that host (load: the least loaded, session: the user is logged in at its keyboard, named: the instructor named it), whether the instructor mode or the debug mode was asked (instructor, debug), result (created, flavor_form, flavor_error, error) and rln, the running lab name of the new project;

• project_open / project_close: the project page, open as long as its events WebSocket is (duration in seconds, code: the WebSocket close code; view=instructor for the instructor view); terminal_open / terminal_close (device, join: the session joined, private) and display_open / display_close likewise; a socket refused before it opens (401, 403, 404) gives a close line with status instead of code;

• eval, state (state), stop, display_start: the HTTP status and the duration;

• denied: a call answered 401, 403 or 429 (method, path), login=- when nobody was logged in; admin: a call of sre-web-tool changing the pool, or an action of the Pool page of an instructor (their login); history: an administrator read the log on the History page (Administrators below; the filters asked come as login_filter, addr_filter, event_filter, host_filter, rln_filter, lab_filter, group_filter (the key of the group, kind:name), since_filter, until_filter, before_filter; view=sessions for the Sessions view, view=statistics for the Statistics view with the period resolved in since_filter / until_filter);

• the presentations (Instructors below): present_start (the instructor’s login, host, rln, audience: the number of logins and groups chosen), present_audience (the audience changed), present_stop, and present_open (a student opened the presented page: their login, the presenter); the project calls a spectator makes keep their own events, their close lines carrying spectator=1. The presence socket every page keeps open, the view the presenter publishes and the pointer frames of their presentation socket are not logged;

• the focus on a student (Instructors below): focus_start (the instructor’s login, host, rln, the student) and focus_stop (the instructor of the focus, the student); the view the two pages share and their sockets are not logged;

• the web conferences (Web conference below): webconf_start (the instructor’s login, webconf = the id of the room, its name, audience = the number of logins and groups), webconf_audience (the audience changed, the owner of the room), webconf_attend (an instructor joining a room), webconf_stop, and webconf_join (a join URL served to a participant: their login, webconf, role, status 302); the listing of the rooms is a keepalive, the page the client comes back to after Leave meeting is not logged;

• the private conversations (Private conversations below): webconf_invite (the instructor’s login, the student asked, the host and rln of the project they asked from, the invitation; status 409 when the student is not connected or busy), webconf_accept (the student’s login, the invitation, the instructor, webconf = the room made; 410 when the invitation was over, 502 / 503 as the server answered), webconf_decline, webconf_cancel (the invitation taken back by an instructor, the student) and webconf_leave (one of the two ending the conversation: webconf, owner); an invitation nobody answers expires without a line.

The client address is the last entry of X-Forwarded-For when the TCP peer is one of trusted_proxies (the Apache and nginx examples set that header), the peer address otherwise: set trusted_proxies in every mode to log the real addresses behind a proxy. The login rate limit ([limits] login_per_minute) counts per logged address too. The log holds personal data (names, addresses, what was done when): keep is its retention, and the file is readable by sre-web and root only (/var/log/sre-web is mode 0750).

In a database. [access_log.database] sends the same events to a MariaDB / MySQL table through PyMySQL (python3-pymysql, suggested by the Debian package rather than installed with it since the feature is off by default, or pip install pymysql in the venv; the file and the journal stay what they are), where the History page of the administrators (Administrators below) reads them:

[access_log.database]
host = "db.example.org"          # or unix_socket = "/run/mysqld/mysqld.sock"
user = "sre_web"
name = "sre"                     # the database; password = "…" in an imported secrets file
keep = 400                       # days of rows kept (purged daily); 0 = forever

One row per line: the time (ts, UTC, millisecond precision), the event, the login (empty when nobody was logged in), the client address, the host, the project (rln), the lab, the status and the duration in columns, the other fields of the line as JSON in details (SELECT login, COUNT(*) FROM access_log WHERE event = 'start' GROUP BY login works as expected). The table (table, access_log by default) is created when it is missing, so the account needs CREATE for the first start; or create it by hand and grant SELECT, INSERT, DELETE only (DELETE for the purge of keep):

CREATE DATABASE sre CHARACTER SET utf8mb4;
CREATE USER 'sre_web'@'%' IDENTIFIED BY '…';
GRANT SELECT, INSERT, DELETE ON sre.* TO 'sre_web'@'%';
CREATE TABLE sre.access_log (
  id       BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
  ts       DATETIME(3)     NOT NULL,
  event    VARCHAR(32)     NOT NULL,
  login    VARCHAR(64)     NOT NULL DEFAULT '',
  addr     VARCHAR(64)     NOT NULL DEFAULT '',
  host     VARCHAR(64)     NOT NULL DEFAULT '',
  rln      VARCHAR(255)    NOT NULL DEFAULT '',
  lab      VARCHAR(255)    NOT NULL DEFAULT '',
  status   VARCHAR(16)     NOT NULL DEFAULT '',
  duration DOUBLE          NULL,
  details  TEXT            NOT NULL,
  PRIMARY KEY (id), KEY ts (ts), KEY login_ts (login, ts), KEY addr_ts (addr, ts), KEY event_ts (event, ts)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

The rows are written by a thread of the portal, in batches, and a request never waits for the server: while it does not answer, the rows wait in memory (queue of them at most, the oldest dropped beyond; one warning in the portal’s log, one line when the writing resumes; the History page says it too) and the portal goes on; a batch the server refuses is written row by row and a refused row is dropped with a warning; the rows still waiting when the portal stops get one last attempt. The rows older than keep days are deleted once a day (DELETE … LIMIT: a replicated server needs a row-based binlog). Every consultation of the History page is a history line, in the file and in the table alike. The table holds the same personal data as the file: keep is its retention, and the account of the portal should be the only one reading it. In the production container the driver must be in the image (or the venv) and the server reachable from it (a local MariaDB over the host network, or its socket bind-mounted).

Retention. The portal keeps keep days of each store while it runs: the rotated files beyond keep go at the next rotation, the rows older than keep days once a day. scripts/web-purge-log.py applies a retention of its own, for a cron job or a timer of the host (run with the Python of the portal, PyMySQL included; it reads web.toml): --days (365 by default), --dry-run, --optimize (the table rebuilt after the deletion, so that the deleted rows no longer occupy its file; a short lock), --files-only / --database-only, --archives-days N (the archives of the evaluations too, below; left alone otherwise); the current file is never touched, a rotated file goes by its modification time. For instance in /etc/cron.d/sre-web:

15 4 * * * root /opt/sre/venv/bin/python3 /opt/sre/scripts/web-purge-log.py --days 365

The archives of the evaluations

Every sre eval writes an archive on the host that ran it (params.archive_dirs, /var/lib/sre/archives: the student’s self-evaluations, the periodic evaluations of the web agent, an instructor’s, the final one of a project closed with eval_before_exit, those of the Qt GUI), and the Watch page reads the newest archive of each running project. With [access_log.database] configured, the portal also collects every archive of every pool host into a second table of that database ([archives] of web.toml: table, archives by default; keep, 0 = the rows are kept forever; poll_interval, 30s), and the Archives page of the instructors (Instructors below) reads it: the results of a project survive its closing, the history of its evaluations is kept, and the results of a student or of a class group can be searched across the labs and the hosts.

The portal pulls, as it does for everything else: every poll_interval the collector asks each reachable agent for the archives of its host it has not taken yet (GET /v1/archives?after=<file name>&limit=50, the token alone: the archives whose name is greater than the last one taken, in the order of the names, which start with the evaluation time; up to ten calls per host and round, so that a host with a backlog of thousands of archives, the first run on a site that already evaluated for months, is imported at about a thousand archives per minute), and queues one row per archive. The agent decodes the archives and sends what the page needs: the project, the lab, the evaluation time, the student’s login, full name, group, e-mail, the host that evaluated, the exam flag and fields, the exo-eval and self-eval totals and marks, the errors and warnings, and every element of the grade list with its title and description as the lab wrote them (in every language of the lab: the browser shows the viewer’s), its grade, maximum, letter, part and bonus flag. The answers of the students are not sent and not stored. The group and the full name an archive lacks (the projects of the Qt GUI) are completed from the directory when the portal reads one (attr_group, ldap_roles / ldap_groups), as on the Watch page.

The reports an instructor downloads from the page (the spreadsheet of sre sheet, the PDF reports of sre outline or their summary spreadsheet alone: Instructors below) are built from these rows, not from the archive files of the hosts, which may be asleep or reinstalled: a row holds everything the two commands read from an archive (the student, the time, the totals, the grade elements with their titles and parts, the number of errors; never the tests nor the answers). The portal turns the rows back into the records the agents had sent and hands them to one agent of the pool, chosen as the lab list is (an enabled, reachable host, up and little loaded first; the next one when it does not answer), which runs sre report (an internal command: the records on stdin, the file on stdout, one report at a time per host, [timeouts] report of agent.toml, 600 seconds) and answers the download the portal streams to the browser. The files are therefore those sre sheet / sre outline would give on the same archives, in the language of the instructor, but for one thing: a student is a login whatever the host of the pool the project ran on, where the two commands tell a login on two hosts apart (Reports, under Instructors below); the elements of an evaluation whose details were too big for the table (over 1 MB) are missing from them. The PDF reports and their summary read the grade elements of one evaluation per student only, the best one (the highest total, the earliest of them): when every evaluation of the selected projects is asked, the portal sends the others as light records (the student, the time, the totals, the error counts: about 300 bytes instead of several kilobytes), read without their details from the database, so that such a request may cover 150 000 evaluations (hundreds of projects evaluated every minute for hours) where the spreadsheet of sre sheet, which shows the elements of every evaluation, stops at 5000. Every download is a report line of the connection log (the kind, the rows ticked, the options, the evaluations and projects sent, full = the evaluations sent with their details, the host that built the file).

One cursor per host, the greatest archive name taken, lives in the portal’s memory and is read from the table itself at the first round after a start (SELECT host, MAX(archive) … GROUP BY host): nothing is collected while the database does not answer, and a restart never imports a host again. The name of an archive is unique in the table (UNIQUE (archive), INSERT IGNORE), so a host whose archive directory is shared with another one, or rows still queued when the portal stopped, only cost a harmless re-insert. An archive rewritten in place (sre re-eval -p '' -d <archive dir>) or named below the cursor (a host whose clock was set back) is not taken again, as sre watch would not see it either: the results of a re-evaluation reach the table through sre re-eval --update-web (Re-evaluated archives below). An agent not yet updated answers 404 and is skipped with one warning until it is.

The table is created like access_log when it is missing (the CREATE privilege, once); or create it by hand and grant SELECT, INSERT, DELETE on it (the same grant as the log’s: the account needs no UPDATE):

CREATE TABLE sre.archives (
  id              BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
  ts              DATETIME(3)     NOT NULL,            -- the evaluation time, UTC
  host            VARCHAR(64)     NOT NULL DEFAULT '', -- the pool host (the name of [[hosts]])
  archive         VARCHAR(300)    NOT NULL,            -- the file name: unique
  rln             VARCHAR(255)    NOT NULL DEFAULT '', -- the running lab name
  lab             VARCHAR(255)    NOT NULL DEFAULT '', -- sre/dhcp.py: the path the student started
  canonical_lab   VARCHAR(255)    NOT NULL DEFAULT '', -- the file really run, symlinks resolved (sre/dhcp.py for s3/r306/dhcp.py -> ../../sre/dhcp.py)
  login           VARCHAR(64)     NOT NULL DEFAULT '',
  fullname        VARCHAR(255)    NOT NULL DEFAULT '',
  grp             VARCHAR(64)     NOT NULL DEFAULT '', -- the student group
  hostname        VARCHAR(64)     NOT NULL DEFAULT '', -- the host the archive names
  grade           DOUBLE          NOT NULL DEFAULT 0,  -- exo-eval scope
  max_grade       DOUBLE          NOT NULL DEFAULT 0,
  mark            DOUBLE          NULL,
  maximum_mark    DOUBLE          NULL,
  errors          INT             NOT NULL DEFAULT 0,
  warnings        INT             NOT NULL DEFAULT 0,
  exam            TINYINT         NOT NULL DEFAULT 0,
  auto_eval_count INT             NULL,                -- the self-evaluations of the student so far
  details         MEDIUMTEXT      NOT NULL,            -- JSON: elements, parts, errors, self-eval totals, metadata
  PRIMARY KEY (id), UNIQUE KEY archive (archive), KEY ts (ts), KEY login_lab_id (login, lab, id),
  KEY login_canonical_id (login, canonical_lab, id), KEY rln_id (rln, id), KEY lab_ts (lab, ts),
  KEY canonical_lab_ts (canonical_lab, ts), KEY grp_ts (grp, ts), KEY host_archive (host, archive)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

Upgrading a table created before canonical_lab (releases up to 0.15.0): the portal probes the columns when it connects and, while one is missing, keeps the rows in memory and warns once (archives database: the table archives lacks a column the rows need …) instead of refusing them; add the column and its keys from the root console before the portal restarts with the new code, then give the old rows their own path as the file run, and run that UPDATE once more after the restart: the rows the old code inserted meanwhile have an empty value (the account of the portal has neither ALTER nor UPDATE):

ALTER TABLE sre.archives ADD COLUMN canonical_lab VARCHAR(255) NOT NULL DEFAULT '' AFTER lab,
  ADD KEY login_canonical_id (login, canonical_lab, id), ADD KEY canonical_lab_ts (canonical_lab, ts);
UPDATE sre.archives SET canonical_lab = lab WHERE canonical_lab = '';
-- to merge the old rows of a symbolic link with their lab, by hand:
-- UPDATE sre.archives SET canonical_lab = 'sre/dns1.py' WHERE lab = 's3/r306/dns1.py';

The new rows carry the file really run as soon as the agents run the new code (an older agent sends none and the row gets its path). The rows go through the same kind of writer as the log’s (batches, the rows waiting in memory while the server does not answer, a refused row dropped with a warning, a last attempt at shutdown) and are purged daily beyond keep days when set; web-purge-log.py --archives-days N applies a retention from a cron job. Expect about thirty rows per project and hour of auto-evaluation (eval_interval_without_exam_mode), a few kilobytes each. The table holds personal data (names, groups, results): the account of the portal should be the only one reading it, and the page is open to the instructors only. Every consultation of the Archives page is an archives line of the connection log, with the filters asked.

Re-evaluated archives

A grading bug found after the evaluations is fixed in the lab file and the archives are graded again with sre re-eval, which replays the lab on the test results each archive holds. The table of the portal was filled when the evaluations ran and does not follow by itself; sre re-eval --update-web makes it follow:

# as root, on the pool host whose /var/lib/sre/archives holds the archives
sudo sre re-eval -s /opt/sre/lab/s3/r306/dns1.py -p corrected -d /root/regraded \
                 --update-web /var/lib/sre/archives/*@@@s3@r306@dns1.py@@@*

The command writes the re-graded archives as usual (-p, -d) and, for each of them, queues the new results on the host: one file per archive in /var/lib/sre/web-updates/ (mode 0700, the files 0600, owned by sre: they name students and grades), holding what the agent sends for an archive — never the answers — under the name of the original archive, which is how the portal knows the evaluation. A later re-evaluation of the same archive replaces the one still waiting. Nothing else is to be configured: no address of the portal, no token, no privilege of the database account.

The portal takes them as it takes the archives. The answer of GET /v1/archives says how many re-evaluations wait on the host (updates); when there are some, the collector reads them in pages (GET /v1/archives/updates?after=&limit=, the token alone), writes each row again in one transaction per page, and tells the agent which ones are settled (POST /v1/archives/updates), which removes them from the host. A row is written with REPLACE under its own id — the INSERT and DELETE privileges the account already has; no UPDATE — so it keeps its place on the Archives page and the links to it. Of the stored row it also keeps what only the first collection knew:

Kept from the stored row	Taken from the re-evaluation
id, ts (the time of the evaluation), host	grade, max_grade, mark, maximum_mark
canonical_lab (the lab file as it resolved then)	errors, warnings and the error list
fullname, grp (the directory’s, at the time)	the elements and the parts of the grade list
the managed detail	the self-eval totals, re_eval_date

The reports downloaded from the page are built from the rows, so they show the new results at once. The Watch page reads the archive files of the running projects, not the table.

• Only the evaluations the table holds are updated; none is added. A re-evaluation whose evaluation is not there yet (collected at the next round, or the database did not answer) stays on the host and is tried again at every round; the agent gives it up, with a line of its log, after a day. Give the archives under their original names: the output of an earlier re-eval and a renamed archive are not queued.

• Run it where an agent runs. The queue is read by the agent of the same host. With an archive directory shared by the pool (archive_dirs), one run on any pool host updates the evaluations of every host: a row is found by the name of its archive, whichever host it came from. On a host without agent.toml the command warns that nothing will hand the updates over.

• Applied at least once. The portal acknowledges after its transaction is committed; a round interrupted in between writes the same row again at the next one, which changes nothing. A database or an agent failing here is warned about once in the portal’s log (archives: <host> (re-evaluations): …) and never stops the collection of the archives.

• The portal logs archives: <host>: N evaluation(s) replaced by their re-evaluation; the Archives page’s database status counts them (collector.updated). A re-evaluation of an archive whose row belongs to another project is dropped with a warning.

• An agent older than this feature never announces the queue and a portal older than it, or one without [access_log.database], never asks for it: the files wait on the host. Whatever waited for more than a day is dropped when the queue is next read by the agent or written by sre re-eval --update-web, which then warns that the portal never took them.

User groups

The Groups page of the instructors (/groups, the link in the header) lists the groups of users the portal knows, of three kinds, each with its members (login, name, department):

• departments: the values of attr_group of [auth.ldap] (departmentNumber) over every person of search_base, the student group the portal has always carried per user (the Group column of the Watch page), listed as groups;

• directory groups: the entries of group_search_base matching groups_filter (posixGroup by default), named by attr_group_name (cn), their members the values of attr_group_members (memberUid): each value is looked up among the people as their login, their uid or their DN (a groupOfNames with member DNs works with attr_group_members = "member"); a value matching nobody but shaped like a login counts as that login, the DN of a nested group is dropped;

• custom groups: the groups the instructors create on the page (a name of at most 64 characters without : nor /, which may not be that of a department or of a directory group, a description), fill (logins typed or picked among the people of the directory, which refuses an unknown login; or every member of another group of any kind at once), empty (one member at a time, or at once every member who also belongs to another group of any kind) and delete. Any instructor may change any custom group; the creator and the dates are shown. A user may belong to several groups.

The directory is read whole, with the service account (or anonymously) of [auth.ldap]: two paged searches of PAGE = 500 entries (the people base of a site exceeds the default size limit of a server), kept directory_ttl and read again on demand (the Refresh button, at most once per refresh_interval: the clicks arriving during a read get that reading, the next ones within the interval the reading just made, so that a row of clicks never dumps the directory once per click). The listing needs uri, search_base and the login attribute of the people: attr_login, or the attribute of a search_filter of the form (attribute={login}) ((supannAliasLogin={login})); the departments need attr_group, the directory groups group_search_base. It works in every authentication mode, proxy mode included. The page says what is not configured, and shows the error of a directory read that failed (the previous reading stays until the next attempt, one minute later).

The custom groups live in two tables of the database of [access_log.database] ([groups] table, user_groups by default, and <table>_members), on iff that section is set. The tables are created when they are missing (the CREATE privilege, once), or create them by hand; the account needs SELECT, INSERT, UPDATE, DELETE on these two tables (UPDATE: a renamed group or a changed description; the other tables keep SELECT, INSERT, DELETE):

CREATE TABLE sre.user_groups (
  id          BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
  name        VARCHAR(64)     CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
  description VARCHAR(255)    NOT NULL DEFAULT '',
  created_by  VARCHAR(64)     NOT NULL DEFAULT '',
  created_at  DATETIME(3)     NOT NULL,
  PRIMARY KEY (id),
  UNIQUE KEY name (name)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
CREATE TABLE sre.user_groups_members (
  group_id    BIGINT UNSIGNED NOT NULL,
  login       VARCHAR(64)     CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
  added_by    VARCHAR(64)     NOT NULL DEFAULT '',
  added_at    DATETIME(3)     NOT NULL,
  PRIMARY KEY (group_id, login),
  KEY login (login)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
GRANT SELECT, INSERT, UPDATE, DELETE ON sre.user_groups TO 'sre_web'@'%';
GRANT SELECT, INSERT, UPDATE, DELETE ON sre.user_groups_members TO 'sre_web'@'%';

The name and login columns are binary-collated: the names and logins are exact and case-sensitive, like the logins everywhere else (a table created with the default collation would find INFO1 for info1). The members of a deleted group are deleted with it, in one transaction; the rows are never purged.

Every group is named by a key kind:name (department:1A, directory:etudiants, custom:TP lundi; a bare name is a department: the links and audiences made before the kinds keep working). The groups of every kind are offered by the group select of the Watch page (the column still shows the department) and by the audience dialog of a presentation: a student is in the audience when one of their groups is chosen, the department by the group of their identity, the others by the memberships the portal knows, read again every 30 s and after every change made on the page (a student added to a presented group is told at once, one removed loses the presentation). Every reading of the page is a groups line of the connection log, every change a group_edit line (action = create / update / delete / add / remove, group, and member (the one member removed), source (the group whose members were added or removed), count (the logins given to an add, the members a removal by source took out) or new_name).

Exams

The Exams page of the instructors (/exams, the link in the header) programs the exams of the web site. An exam is a title, an audience (logins and/or groups of any kind of the Groups page: a department, a directory group, a custom group), the labs to do (the exam-only labs, whose names hold _EXAM_, _OLD_, _DRAFT_ or _TESTS_, are offered here and nowhere else; each lab may name a preset of its Flavor class, as sre set-exam --labs lab:preset does), a duration and/or a window (start_after, end_before; no start time = the exam starts when created, for each student as soon as they log in), two delays before the start — the lead (the other projects of the students are hidden and a countdown appears) and the pre-start (their exam projects are started in the background) —, the networks the students’ browsers must come from (none = anywhere; the address is the client’s, through trusted_proxies) and the evaluation interval of the agents during the exam. The defaults of the delays and of the interval are those of [exams].

Unlike the exam mode of the Qt GUI (sre set-exam, exam.json, which locks a whole workstation and keeps a pool host out of the pool), an exam of the web site binds its students only, wherever their projects run: the portal computes the phase of each student and enforces it on every request, the agents apply the exam rules to the exam projects alone. Instructors are never bound, even when listed in the audience.

What a student sees, in order:

• scheduled: nothing changes before the lead window.

• waiting (the lead window, 10 min by default): a bar on every page counts down to the start; their other projects are hidden and refused (the Open page too); from the pre-start (5 min before) their Projects page starts the exam projects in the background, one at a time (the start slot and the placement of a normal start; a lab asking a flavor form shows it), and shows them without an Open link.

• starting: the start passed (or the exam has no start time), the exam projects are being started; the clock of the student starts when every lab of theirs has a running project.

• active: the countdown to their end; the Projects page lists the exam projects alone (a single one is opened by itself at the start); the project page has no Close button, no Evaluation tab nor Apply Configuration tab (as in the Qt GUI); the agent evaluates the project every eval_interval of the exam. Each student has an individual clock: their end is min(started + duration, end_before) plus the minutes an instructor adds to them; an End now ends everybody’s exam, or a selection’s, at once.

• ended: the end banner (That’s all folks) until the student presses Continue; the agent has cut the terminals and the X desktop, run the final evaluation and refuses every change (the answers included). After Continue the usual pages are back; the exam projects stay hidden from the student (and refused) until they are closed: close_after after the end by the agent (the idle timeout of an exam project counts from its end), or by the instructor (Close projects on the page, or the Watch page; the page stops them in the background, eight at a time, and shows the progress under the title of the exam; a second click while a close runs changes nothing).

• A student connecting from an address outside the networks of their exam gets the bar in red naming their address, and no project at all (neither the exam’s nor the others) while the exam binds them.

A student in several exams at once follows the one started first. The phases are pushed to the open pages through the presence socket ({"t": "exam", "id", "phase"}: the pages read GET /api/exam then) and the countdowns use the portal’s time.

On the page the instructor sees, per exam, every student the portal can name (the logins, the members of the groups it knows, those it has seen), connected or not, with their address, their phase, their clock, their extra minutes and their running exam projects (a link to each: the instructor’s view of the student’s project, the Evaluation tab included), and can, for everybody or for a selection (the checked rows): add or take minutes (for everybody the duration and the end time move together and an End now is lifted; for a selection their extra minutes, cumulative), End now, change the labs (the browsers start the new ones; the projects of a lab dropped stay as plain projects), Close projects, Start now (the start time set to now), Edit (every field; the students keep their clocks and adjustments) and Delete (the projects become plain projects, or are closed with the checkbox). The listing shows the scheduled, the running and the past exams.

The exams live in two tables of the database of [access_log.database] ([exams] table, exams by default, and <table>_members), on iff that section is set (the page says so otherwise): an exam survives a restart of the portal, the clocks of the students included, and the agents are told again what they missed. The tables are created when they are missing (the CREATE privilege, once), or create them by hand; the account needs SELECT, INSERT, UPDATE, DELETE on the exams table (an edited exam) and SELECT, INSERT, DELETE on the members one (its rows are replaced):

CREATE TABLE sre.exams (
  id          BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
  title       VARCHAR(200)    NOT NULL,
  created_by  VARCHAR(64)     NOT NULL DEFAULT '',
  created_at  DATETIME(3)     NOT NULL,
  updated_at  DATETIME(3)     NOT NULL,
  start_after DATETIME(3)     NULL,
  end_before  DATETIME(3)     NULL,
  duration    INT             NULL,
  ended_at    DATETIME(3)     NULL,
  details     MEDIUMTEXT      NOT NULL,
  PRIMARY KEY (id),
  KEY start_after (start_after),
  KEY created_at (created_at)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
CREATE TABLE sre.exams_members (
  exam_id       BIGINT UNSIGNED NOT NULL,
  login         VARCHAR(64)     CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
  first_seen    DATETIME(3)     NULL,
  started_at    DATETIME(3)     NULL,
  ended_at      DATETIME(3)     NULL,
  dismissed_at  DATETIME(3)     NULL,
  extra_minutes INT             NOT NULL DEFAULT 0,
  address       VARCHAR(64)     NOT NULL DEFAULT '',
  details       TEXT            NOT NULL,
  updated_at    DATETIME(3)     NOT NULL,
  PRIMARY KEY (exam_id, login)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
GRANT SELECT, INSERT, UPDATE, DELETE ON sre.exams TO 'sre_web'@'%';
GRANT SELECT, INSERT, DELETE ON sre.exams_members TO 'sre_web'@'%';

duration is in seconds; details holds the audience, the labs, the delays, the networks and the interval (JSON), the members’ details their labs and their exam projects. The times are UTC, truncated to the millisecond like the rows of the connection log. The past exams are kept keep days (0 = forever), their members with them.

The agents: an exam project is started by the portal with an exam object (POST /v1/projects: the id and title of the exam, the start and end of the student’s clock once known, the evaluation interval) and a privileged sre start run under the student’s login (LOGNAME), which accepts the exam-only labs and the flavor presets (--set-flavor-name); the agent keeps the object in the project’s state file and in the rows of its listings (exam), evaluates the project at the interval of the exam once its clock runs, never reaps it before the end, refuses the owner’s self-evaluation, states and close during the exam and every change after the end, and at the end cuts the terminals and the display, runs the final evaluation (sre --user eval: the exo-eval scope, like sre end-exam) and tells the open pages ({"t": "exam", "ended": true} on the events socket). The portal updates the object with PUT /v1/projects/{rln}/exam when the clock starts or the instructor adjusts the student (every 5 s until the agent has it), and drops it with DELETE …/exam when the exam is deleted or the lab is no longer the student’s. The answers metadata of an exam project carry the exam fields of the Qt GUI’s exam mode (exam_mode, exam_started_at, exam_duration, exam_time_remaining), so the archives carry the exam flag and the Archives page filters on it. The exam-only labs of the pool are listed by GET /v1/labs?all=1 (instructors), a privileged sre list --with-titles --include-exam-only.

Connection log: every change of an exam is an exam_edit line (action = create / update / delete / end / adjust / labs / close, exam, count of students, minutes), a student dismissing the end banner an exam_dismiss line, and the start lines of the exam projects carry exam; the polls of the page and of the student’s exam are keepalives.

Limitations: the session records (record_sessions) are not archived by the agents (the lab’s attribute decides whether records/ exists, sre stop deletes it); the archives of a web exam hold no exam_json document (the exam flag comes from the answers); anonymous mode has no stable logins to schedule; a page loaded before the portal was updated has no exam script until it is reloaded (the server still refuses what it must).

Web conference (BigBlueButton)

An instructor talks to a student, a group or the whole class without leaving the site: the portal opens rooms on a BigBlueButton (BBB) server and shows its client to the participants in an iframe over the pages of the site, while everybody goes on working on their projects. Nothing is installed on the pool hosts; the portal needs the API of the server, the browsers the server itself.

[bbb]
url = "https://bbb.example.org/"     # the server (its bigbluebutton/ or bigbluebutton/api/ path works too)
secret = "..."                        # the shared secret of its API: bbb-conf --secret
# checksum = "sha1"                   # sha1 (default), sha256 or sha512: supportedChecksumAlgorithms of the server
# welcome = { fr = "Bienvenue", en = "Welcome" }   # the welcome message of the client (the server's by default)
# record = false
# mute_on_start = false
# duration = 0                        # "2h": the server ends a meeting after that; 0 = unlimited
# max_participants = 0
# poll_interval = "10s"               # the participants and the ended meetings are read that often
# private_rooms = true                # Talk privately on a student's project (Private conversations below)
# [bbb.userdata]                      # the client settings of every join (userdata-bbb_*, the BBB API reference)
# bbb_skip_check_audio = true
# [bbb.private_userdata]              # the client settings of the joins of a private conversation, over the others
# bbb_listen_only_mode = true

The instructors: the Webconf button of the site bar opens the dialog of the rooms: those open on the portal, with their owner, the size of their audience, who is connected to the site and who is in the room (read from the server every poll_interval), each with Join (this browser shows the room, as a moderator: any instructor may join any room), Audience… and End (any instructor); and a New room form: a name (the instructor’s by default) and an audience chosen as for a presentation (the connected users one by one; the groups: departments, directory groups, custom groups; a member of a group connecting later is put in the room too). Unlike that of a presentation, the audience of a room may be empty (None then Choose): the room is then one of the instructors alone. A student is in at most one room at a time (the newest one including them); an instructor sees the room they created or joined last.

The participants: the client appears in a floating window over the page (dragged by its title bar, resized by its corner), with buttons to minimize it to its title bar, to dock it into the Webconf tab of the project page, to show it full page (a bar above it comes back), to pop it out to a separate browser window, and to leave (×). How it is shown and where the window is are remembered by the browser: the next page shows the conference the same way (in a window where there is no project tab). A participant who left, or whose client is in a separate window, keeps a pill in the site bar to rejoin or to bring it back; the Leave meeting of the client itself leads to that pill. Every page of the site loads the client again (a new join of the same room: a few seconds, the audio to join again), which a site of several pages cannot avoid: the separate window does. Two tabs of the same user never join twice: one holds the client, the others show a pill to bring it there.

The server: the meeting is created when the room is (a server that refuses or does not answer leaves no room: the dialog says so); a join is a URL signed for the user (fullName = their name, userID = their login, the role by their instructor flag: a moderator for an instructor, a viewer otherwise, the bbb.userdata settings), served by the portal as a redirect (GET /api/webconfs/{id}/join, the src of the iframe): the secret never leaves the portal. A meeting nobody joined expires on the server after a few minutes (meetingExpireIfNoUserJoinedInMinutes): the portal creates it again at the next join. A meeting everybody left expires about a minute later (meetingExpireWhenLastUserLeftInMinutes), like one a moderator ended from the client: the room ends on the portal at its next poll, every participant told. End in the dialog ends the meeting on the server. The rooms live in memory: a portal restart forgets them (the meetings expire on their own, the clients already in them stay until then).

Prerequisites: the portal reached over HTTPS, with public_url set (the logoutURL of the meetings, where the client comes back after Leave meeting, is built from it: /webconf/left); the server reachable from the portal (its API) and from the browsers (the client, over HTTPS); the nginx of the server must not forbid framing by the portal’s origin (an X-Frame-Options header or a frame-ancestors directive on the client would leave the frame empty: the separate window is the fallback); the pages of the portal allow the server’s origin in the frame-src of their Content-Security-Policy, nothing else. BBB 2.4 at least (the role of the join; the passwords are still passed for older servers). Safari may refuse the camera in a cross-origin frame: pop the client out.

Private conversations

An instructor who opened a student’s project (the helper view) may talk with that student alone, voice and text. BigBlueButton has no private audio inside a meeting (one audio bridge per meeting; its breakout rooms are made by hand in the client and open in another browser tab), so a private conversation is a second meeting of the two, which both browsers show in place of the class conference for as long as it lasts: nobody is ever in two meetings at once.

• Talk privately, in the bar of the project after Focus student on project (shown with [bbb] unless private_rooms = false), invites the owner of the project. Nothing else happens yet: both stay in the class conference, no meeting is created. The button reads Calling… (cancel) and takes the invitation back; an invitation nobody answers expires after a minute. The student must be connected to the site, and neither invited by somebody else nor in a private conversation already (the instructor is told so).

• The student gets a banner across the top of every page of the site they have open (above a presentation, inside an open dialog), with a short chime and a mark in the title of the page: the name of the instructor, would like to talk with you, and two buttons, Join and Not now. Not now tells the instructor; nothing opens a microphone before Join.

• Join creates the meeting and sends both browsers to it: the window of the conference (at the place and in the mode of the class conference: window, tab or full page) now shows the private room, titled Private — and the name of the other one, its title bar in its own colour. The joins go straight to the microphone, without the echo test or the choice of Listen only (bbb_auto_join_audio, bbb_listen_only_mode = false, bbb_skip_check_audio, bbb_hide_presentation_on_join; [bbb.private_userdata] overrides them, e.g. bbb_listen_only_mode = true to get the choice back); the meeting is never recorded and starts nobody muted, whatever [bbb] says for the rooms. The chat of that meeting is the text channel.

• Only the tab where the user clicked — Talk privately for the instructor, Join for the student — loads the client: the other tabs, and any other browser the same user is logged in from, leave the class conference and show the pill (Bring here moves the conversation). A browser left open somewhere never opens its microphone by itself.

• The end: the × of the window, End conversation in the bar of the project, or the Leave meeting of the client, by either of the two — leaving a conversation of two ends it; End in the Webconf dialog, by any instructor (the dialog lists the conversation, nobody else may join it or see its audience); and the server itself, a minute after the instructor is gone (a closed browser: endWhenNoModerator). Both browsers then show the class conference again, in the mode and at the place it had; an instructor comes back to the room they were in.

• An instructor already talking with a student may invite another one: the first conversation ends when the second one is accepted, and the instructor goes from one to the other. Joining another room from the Webconf dialog ends the instructor’s conversation.

Limits: each switch reloads the client (a few seconds, twice per conversation), and both are out of the class conference meanwhile — the other students cannot reach the instructor there. A class conference popped out to a separate window is closed when the conversation starts (by the tab that opened it) and comes back in a window of the page; when that tab was closed since, the site cannot close the separate window: the user is then in both meetings until they close it. The invitations live in memory like the rooms.

The connection log gets the webconf_* events (The connection log above). Tests: tests/test_bbb_client.py, tests/test_web_webconf.py (the registry on a fake client, the portal in front of the fake server of tests/fake_bbb.py), tests/test_web_webconf_browser.py (headless Chrome: the rooms, then the private conversations — the banner, the switch and the return of two pages, two tabs of one user holding one client).

Securing the pool (mutual TLS)

The portal and the agents can authenticate each other with certificates of one private CA, on top of the token: the agent serves TLS and requires a client certificate of that CA, the portal verifies the agents against the same CA and presents its client certificate. scripts/web-tls.sh makes the files with openssl:

scripts/web-tls.sh init-ca /root/sre-tls                      # ca.key, ca.crt (10 years)
scripts/web-tls.sh agent /root/sre-tls a01 IP:10.0.0.11 DNS:a01.example.org   # a01.key, a01.crt
scripts/web-tls.sh portal /root/sre-tls                       # portal.key, portal.crt

On each pool host, agent.toml gets its own certificate and the CA (tls_cert = ".../a01.crt", tls_key = ".../a01.key", tls_ca = ".../ca.crt", files readable by root only) and, as a second fence, allowed_clients = ["<address of the portal>"]. On the portal host, web.toml gets [pool] ca = ".../ca.crt", client_cert = ".../portal.crt", client_key = ".../portal.key" (readable by sre-web), and every [[hosts]] URL becomes https://<name or address in the certificate's SANs>:8451. The portal then logs TLS to the agents: CA …, client certificate … and each host as reachable; an agent without the right certificate, or a client without one, fails at the handshake, before any request.

A whole site at once (scripts/make-web-hosts.sh)

scripts/make-web-hosts.sh writes everything the previous section asks for, for every pool host of a site, from a list of its machines:

IP_MAC_FILE=ip_mac.txt OUT_DIR=/root/sre-pool REGEXP='^([a-z])[0-9]+$' ROOMS="a h" DISABLED_ROOMS=h \
    scripts/make-web-hosts.sh

The list has one mac;ip;name line per machine (e4:54:e8:d1:6c:1c;10.0.128.1;a01: the MAC address for Wake-on-LAN, the IPv4 address the portal reaches its agent at, the name). A machine gets an agent when its lower-cased name matches REGEXP; the first group of the expression, when there is one, is its room, so that ROOMS selects the rooms joining the pool (empty: every qualified machine) and DISABLED_ROOMS those whose hosts start with enabled = false (polled, no project placed there until sre-web-tool enable). These five variables are required and have no default; AGENT_PORT (8451), MAX_PROJECTS (25), POWER (power = true in the entries and allow_power_off = true on the agents), DOMAIN (a DNS:<name>.<domain> SAN too), PORTAL_ADDRESS (allowed_clients of the agents), INSTRUCTOR_ROOMS (the rooms whose hosts get instructors_only = true) and INSTRUCTOR_MACHINES (the names of the hosts that get it on their own, whatever their room), TLS_DIR_ON_HOSTS (/etc/sreweb/tls, the paths written in the files) and CA_KEY / CA_CRT (both set: an existing CA copied into OUT_DIR instead of a new one, so that the hosts of a new OUT_DIR are signed by the CA the portal already knows; refused when OUT_DIR holds another CA) are optional.

In OUT_DIR (private keys and tokens: outside any repository) it writes, through web-tls.sh, the CA and the portal’s client certificate once, then per host a key and a certificate (SANs IP:<ip> and DNS:<name>), a token in <name>.token, and:

• hosts.toml: the [[hosts]] entries, one room after the other, with the three [pool] TLS lines of web.toml in its header; a file of web.d/ (import = ["web.d"]) or pasted at the end of web.toml;

• agent-<name>.toml: the host-specific lines of that host’s agent.toml (listen = "<ip>:8451", token, tls_*, and allowed_clients / allow_power_off when asked), to complete with the example file, whose own listen and token lines are then removed (a key given twice is refused).

Copy <name>.crt, <name>.key and ca.crt to TLS_DIR_ON_HOSTS of each host, and ca.crt, portal.crt, portal.key to the portal’s. Every run computes hosts.toml and the agent files from the variables of the moment, so the rooms, the disabled or reserved hosts may change at any time, and compares them with the previous run: a file whose content did not change is left as it is, the others are replaced, what changed is printed per host (enabled, instructors_only, allow_power_off…) with the new hosts and those no longer selected, and the last line says whether hosts.toml and which agent files are to deploy again; the CA and the portal’s certificate are made once. The hosts’ keys, certificates and tokens are made anew on every run, to deploy again, unless the script is run with --keep: the files already there are kept and only those of the new hosts are made (a kept certificate that no longer names the address of its host is reported), so a room is added, or a host disabled or reserved, without touching the hosts already deployed.

A pool of workstations

The pool hosts are typically classroom workstations: off when nobody needs them (a nightly cron shuts them down), woken by Wake-on-LAN, and sometimes used at the keyboard by a student. The portal gives each host a state from the status of its agent, and sre-web-tool list (or the Pool page of the instructors) shows it:

State	Meaning	New projects
off	the agent does not answer	no
waking	a wake was sent less than wake_timeout ago	no, but a start answers “a host is starting” (503, retried by the Open page)
up	the agent answers, the host is free	yes
occupied	the agent answers, the host is used locally (occupied_check)	only the new projects of the users logged in there (follow_local_sessions); the web projects already running there go on
exam	the agent answers, exam.json is on the host	no
draining	the portal asked the host to power off, or sre-web-tool drain	no; powered off once it is empty
disabled	sre-web-tool disable, or enabled = false	no; the projects already running there go on (listed, usable, evaluated and closed as usual), the host stays polled and is never powered off by the portal — drain is the way to empty a host and shut it down

Placement. A new project (POST /api/projects) goes to one host, chosen by the portal from the last status poll of every agent (poll_interval) without asking them again. The candidates are the hosts up (plus, for a user logged in at the keyboard of an occupied host, that host: Occupied hosts below), below their max_projects, not reserved for the instructors when the caller is a student, and whose agent has no restart pending (sre-web-tool restart: the agent refuses new projects until it is back). Among them: the host of the user’s own session first, then the lowest projects / max_projects, then the lowest one-minute load, then the name, so that equal hosts are always settled the same way. The project count of the chosen host is raised at once and a reservation is kept until the start is over and a status poll sent after its end has answered: a burst of simultaneous starts is spread over the pool instead of piling onto the host the last poll showed emptiest, and sre-web-tool list may count a project one poll before the agent does. Without any candidate the start is refused with 503 and a retry_after the Open page obeys: “a host is starting” (30 s) when a host the caller may use is waking or restarting, “no host available for a new project” (60 s) otherwise. An instructor naming a host on the Open page skips the ranking: the host must exist (404), be open to them (403 for a student), be available to them (503 “not available”) and have room (503 “full”). The start line of the connection log says why the host was chosen: placement=load, session or named.

Waking. A host with a mac gets a magic packet (UDP port 9 to wol_broadcast: the portal must be on the hosts’ network, or the routers must forward the directed broadcast); a host with a wake_command gets that command instead (ssh relay wakeonlan {mac}, an IPMI or VM call…). The portal wakes one sleeping host, in configuration order, whenever the free project slots of the hosts up (plus the capacity of those waking) drop below spare_projects, and keeps min_hosts_up hosts awake during active_hours: the floor is re-established in the morning and not fought at night after the cron. A host still silent after wake_timeout is retried after 1, 2, 4… up to 30 minutes, so a machine that is really down is not hammered. The agent must be an enabled systemd unit on every host (boot → agent up → the portal sees it).

Occupied hosts. The agent reports occupied when a local login session exists (loginctl, who as a fallback) or when projects of the Qt GUI run on the host; the portal places nothing there and never powers it off. The web projects already running there continue. One exception, follow_local_sessions (true by default): the user logged in at the keyboard of a pool host gets their new projects on that host, whatever its load, so that a student working at a classroom workstation uses their own machine through the web too. The agent names the sessions in its status (session:<login>, the OCCUPIED column of sre-web-tool list; its occupied_check must include the sessions) and the portal matches them against the login of the identity, which must therefore be the Unix login of the workstation (the directory uid: AuthLDAPRemoteUserAttribute uid in the Apache example; a guest-… pseudonym never matches). The rule overrides the load and the occupation only: when the host is full (max_projects), in exam mode, disabled, draining or reserved for the instructors, the normal placement applies, and an instructor naming a host on the Open page is obeyed (they may name the host of their own session). The start line of the connection log says placement=session. follow_local_sessions = false restores the strict rule: nothing goes to an occupied host.

Reserved hosts. A host with instructors_only = true never takes a student’s project: an instructor’s goes there when they name it on the Open page (the Host select lists the reserved hosts with their state and their projects) or by the automatic placement when it is the least loaded host they may use. It is outside the elastic pool: its free slots count neither as spare capacity nor towards min_hosts_up, the portal never wakes it by itself (sre-web-tool wake, or its own boot; an instructor finds it greyed while it is off) and powers it off when idle like any host with power = true, whatever the floor and the spare. list shows (instructors) after its state. A student logged in at the keyboard of a reserved host is still placed elsewhere: follow_local_sessions never overrides the reservation.

Going down. When the agent’s unit stops because the system halts (the nightly cron, a poweroff), the agent closes the projects it manages as a Close would (the last evaluation, then sre stop), concurrently, within TimeoutStopSec; the browsers see “project closed”. The unit is ordered after remote-fs.target and, on a checkout, requires the mount of /opt/sre (RequiresMountsFor): it is stopped while sre and the archive directories are still mounted. Unordered, systemd tries to unmount /opt/sre while the agent is still closing a project: the unmount fails (device is busy) and is not tried again, the network goes down with the NFS mount still there, and the machine never powers off (it then answers neither ping nor Wake-on-LAN until somebody presses its power button). What is left on the host afterwards (the projects of the Qt GUI, a project not closed in time) is removed by the unit sre-wipe-at-shutdown.service when it is enabled (sre wipe after the agent has stopped and before Docker does; see Stopping the projects when the machine halts in the installation page), so that no container is running when the system powers off. With power = true on the host and allow_power_off = true in its agent.toml, the portal powers off an idle host itself: idle_host_timeout (0 by default: only the cron does it), above min_hosts_up and as long as the spare capacity stays, or at once with sre-web-tool poweroff.

Pre-loading the images. A project whose Docker images are not on its host yet waits for their download at its start, and a room of workstations pulling them at the same time takes the whole bandwidth of the site; the boot unit sre-preload-images.service (see Installation) only spreads the pulls over a random delay. On the pool, the agents pull and the portal lets a few hosts at a time do it:

• a host asks: its agent starts with [preload] at_start = true in agent.toml (the boot of the host, a restart of the agent), or sre-web-tool preload NAME | --all | --failed or the Preload images button of the Pool page ask for it by hand (after new labs or a new image version). The agent then says pending in its status and pulls nothing;

• the portal gives turns: at every poll, as many waiting hosts as [pool] max_preloads (2) leaves free get POST /v1/preload/start, the one that waited longest first; the agent then runs sre preload-images --progress on its lab directories ([preload] paths) and reports the image it pulls. A request by hand goes through the same queue: the number of hosts downloading never exceeds max_preloads, whoever asked;

• every host that asks takes a turn, even with every image already there: its docker pulls only check the registry, a few seconds, and an image pushed again under the same tag is fetched.

sre-web-tool list shows (preload queued), (preload 2/5) or (preload failed) after the state, the Pool page the same as flags (the image being pulled, or the error, as tooltip), and sre-web-tool status the count of each. A preload that fails (failed: an image that does not exist, the registry unreachable; the error names the images) is not tried again by itself: sre-web-tool preload --failed asks again. A disabled or occupied host is served like the others — its local users start labs too —; a host is not given a turn while projects of an exam of the web site run on it, while it is draining or its agent restarting, and an idle host waiting for its turn is not powered off by idle_host_timeout. A stop or a restart of the agent stops the preload in progress (Docker keeps the layers already there): with at_start the agent started next asks again, a request made by hand is forgotten. Nothing is preloaded without the portal (the hosts stay pending; the images are then pulled by the first start that needs them, as without any preload) nor with max_preloads = 0; the agents older than this feature are left alone. On a host whose agent has at_start, leave sre-preload-images.service disabled: both would pull.

Nothing asks again on a timer. A host that boots every day checks at each boot; one that stays up does not notice new labs or a new image version until its agent restarts (sre-web-tool restart --all after a sync makes every at_start agent ask again) or a preload is asked by hand, and a failed one stays failed. A cron job on the portal’s host makes the check periodic, through the same queue (max_preloads hosts at a time):

0 3 * * *  sre-web-tool preload --all

Restarting the agents. After an update of the tree the agents run the old Python code until their unit is restarted: sre-web-tool restart NAME, or restart --all for every reachable host (whatever its state), tells each agent to restart itself (POST /v1/restart). The agent answers at once, lets its running sre commands end (a start, an evaluation; no new project is accepted meanwhile), then runs systemctl reset-failed and systemctl --no-block restart on its own unit (sre-agent or sysreseval-agent, read from /proc/self/cgroup; restart_command in agent.toml replaces that for an agent run otherwise). The projects stay (close_projects_on_stop = "shutdown": a restart is not a shutdown; with always they are closed), the open terminals and virtual desktops are lost, the project pages reconnect by themselves. The stop is quick: the agent closes the project pages’ WebSockets itself (aiohttp would otherwise wait a minute for them) and KillMode=mixed ends what is left. list shows (restart) after the state until the portal sees the agent’s uptime reset (or gives up after two minutes with a warning in its journal); the host takes no project meanwhile and keeps its drain. --wait makes the tool wait for the agents to be back (--timeout, 180 s by default), so a deployment script can chain it after the sync. A DO_NOT_START_AGENT marker written since the start is honoured (that agent stops); code that no longer starts leaves the unit failed after five attempts (journalctl -u sre-agent, then systemctl reset-failed). The portal itself is restarted by hand (systemctl restart sre-web; Restarting the portal below).

By hand. sre-web-tool … (sbin/sre-web-tool of a checkout, /usr/sbin/sre-web-tool of the sysreseval-web package; python3 -m sreweb.tool from the sources) talks to the running portal through its administration API ([admin] token, loopback only by default; --config, --url, --token override the configuration):

sre-web-tool list                 # one line per host: state, projects/max, load, terminals, occupied by, uptime
sre-web-tool status               # spare capacity, hosts awake vs the floor, active hours, hosts waking, the state file
sre-web-tool add a03 --url https://10.0.0.13:8451 --token … [--mac … | --wake-command …] [--max-projects 25] [--power] [--instructors-only]
sre-web-tool remove a03 [--force] # refused while the host has projects (--force leaves them to the agent's idle reaper)
sre-web-tool disable a03 | enable a03   # disable: no new project on the host, the running ones go on until their users close them
sre-web-tool wake a03 | poweroff a03 | drain a03   # drain: no new project, powered off once the last project is closed
sre-web-tool restart a03 | --all [--wait] [--timeout 180]   # the agents restart themselves (new code), the projects stay
sre-web-tool preload a03 | --all | --failed   # the hosts pull the Docker images of the labs, max_preloads of them at a time

list shows config or added per host; a config host removed by hand comes back, enabled, with enable. --json gives the raw rows. The instructors have the same actions on the Pool page of the portal (/pool, see Instructors below), except add and remove, which stay with the token.

Restarting the portal. What is changed by hand, with the tool or on the Pool page, is kept in [admin] state_file (pool.json, written at every change through a temporary file renamed over it) and merged with the [[hosts]] of web.toml at every start, so that a restart of the portal leaves the pool as it was:

Kept	How
the hosts added and removed by hand	added (the whole entry, its token included: the file is 0600), removed
a host enabled or disabled by hand	enabled / disabled: the hosts whose flag differs from the enabled of their [[hosts]] entry, in both directions (a host of DISABLED_ROOMS enabled for a class stays enabled)
a drain	drain, with the time it was asked; over when the host is gone, or was booted again while the portal was stopped
a wake in progress, the pause after failed wakes or after a power off, a power off or a restart sent	clocks (written within one poll and when the portal stops); each is dropped at the start when too old to mean anything

The file only holds what differs from the configuration, and an edit of the configuration wins: a flag set by hand holds for as long as the enabled of that host in web.toml is what it was then. Once the file says the same (the room taken out of DISABLED_ROOMS, for instance) the entry is dropped at the next start, and a later edit of web.toml is followed again; so is the entry of a host that is no longer in the pool (one line of the journal each). The start logs pool state read from …: 0 removed, 0 added, 21 enabled and 1 disabled by hand, 0 draining.

A state file that cannot be written (a read-only directory, a container without its volume) would lose all of this at the next restart: the portal says so in its journal at the start and at every failed write (pool state NOT saved in …), sre-web-tool status prints pool state: NOT SAVED in … instead of the path, and the Pool page shows it in red above its table. In a container, /var/lib/sre-web must be a mounted directory the sre-web user may write; the systemd units provide it (StateDirectory). A file the portal cannot read (edited as another user, or no longer a JSON object) is not written over: it is renamed pool.json.unreadable, said in the journal, and the pool starts from the configuration; edit the file with the portal stopped and leave it to the sre-web user.

Not kept: the idle clock of idle_host_timeout (it counts again from the start), and what is listed under Limitations (the presentations, the focuses, the open sockets: the pages reconnect by themselves, a terminal with its Reconnect button). The exams, the custom groups, the connection log and the archives are in the database; the sessions are in the browsers’ cookies (secret).

What the users see

• Header: the SRE logo and, with [site] in web.toml, the institution’s logo and name next to it, on every page.

• Login: the directory account, or (anonymous mode) a display name and a language; in proxy mode the web server in front has already asked, there is no login page.

• Projects: the user’s running projects on the whole pool (Open, Close), “Open a lab” (the lab tree with its titles, in the order the titles.json of each directory sets, by file name, by title in the language of the page or by hand, the sub-directories first, the same tree as the Qt GUI’s; the Flavor form when the lab has one; the progress of the pulls and deploys); with no running project the page shows the SRE logo in large, as the Qt GUI does when no project is open. An instructor’s cards say the mode of each project (Standard / Instructor mode / Debug mode, the words of the Open page), the two non-standard modes also as a coloured suffix after the title. Logout asks what to do with the running projects: close them (each as a Close does: the last evaluation when the lab asks for it, then sre stop; the dialog waits for every answer) or leave them running; with no running project the button logs out at once.

• A project: the tabs of the Qt GUI. The schema is drawn by the agent (graphviz) with the same options (lines, spacing, order, icons); the questions and the forms are the same; the evaluation table has the same parts, subtotals, bonuses and colours; the terminals are xterm.js terminals on sre connect: every Connect button of the Machines and Networks tabs opens a new one on its device in a floating window over the page (like the external xterm of the Qt GUI: gw, gw (2), …; the window stays while the tabs change, is dragged by its title bar, resized by its bottom-right corner and closed by its ×, and takes the font size and the colour scheme of the Terminals tab), the two selects of the Terminals tab open one on any device (the machines open to the students and the managed switches) as a sub-tab or in a window, the ⧉ of a sub-tab moves its terminal into a window and the dock button of a window brings it back as a sub-tab (the same shell goes on), the × of a sub-tab or of a window closes it, terminals_per_device caps them; the Log tab appears for debug projects. Export Schema to PDF downloads the drawing of the Schema tab with its current options (never the hidden machines), rendered by the agent as the GUI’s File → Export Schema to PDF does, unless the lab sets export_schema_pdf = False; the icons need rsvg-convert (librsvg2-bin) on the pool host, graphviz shapes otherwise. The Kathara export is not offered on the web. A terminal is a session shared by its viewers: the live sessions of the project that are not on the page are listed under the bar with a Join button (the same shell for everybody, the last 64 KiB of output replayed to the newcomer, the pty sized to the smallest window), a sub-tab or a window shows how many viewers its session has, and a session ends when the shell exits or when its last viewer closes it with its ×. A viewer that just goes away (the page closed, reloaded or cut, another project opened) leaves the session running for terminals_linger of the agent (120 minutes by default), listed idle meanwhile, and the page coming back to the project finds the user’s own terminals again, sub-tabs and windows where they were on that browser, with the last 64 KiB of output (a page the browser kept in its back/forward cache, brought back by the Back button, joins its sessions again by itself); after a disconnection, Reconnect joins the same session again.

• A presentation: when an instructor presents a project to the class (Present below), the page of every chosen student is covered by that project, read-only: every tab is there but nothing can be done on it, and the tab shown, the terminals open (the instructor’s shells, in their exact size, at the largest font that fits; the instructor’s floating windows float on the student’s page too, at the same relative place and size), the question selected, the drawing options of the schema, the instructor texts and the virtual desktop (view-only) follow what the instructor does; so does the position inside the tab: the scroll of the text, of the question (its description and the list of titles), of the tables and of the log (the line at the top of the instructor’s pane is at the top of the student’s, whatever the size of their window or their font size), and the pan and zoom of the schema. The instructor’s mouse pointer is drawn on the student’s page too, an arrow carrying the instructor’s name, and what the instructor selects — in the texts, the question, the tables, the log, the schema, or in a terminal — is highlighted there (in a terminal: selected in the student’s read-only terminal, in lavender); both are placed on the element, the word or the terminal cell the instructor’s mouse is on, never at the same pixel, so that a student with another window size, font size or word wrap sees the pointer on the same word. The Evaluation tab shows the table of the instructor’s page as it is (the letters of their own self-evaluation included), again after each evaluation. The title bar of the presented project is coloured (lavender) and names the instructor presenting it. The student’s own page is untouched underneath, terminals included: when the presentation ends it is back as it was.

Instructors

Who is an instructor is decided by the directory, the role being the OR of the tests configured in web.toml (a test whose key is empty is skipped):

• instructor_groups: the user is a member of one of these groups (DNs), through the memberOf attribute of their entry or, with group_filter = "(member={dn})", a search of the groups under group_search_base (search_base without it);

• instructor_filter: the user’s own entry matches this LDAP filter (a BASE search on their DN), "(gidNumber=2000)" by default — the primary group; a site with the memberof overlay may write "(memberOf=cn=teachers,ou=groups,dc=example,dc=org)"; "" disables the test;

• group_search_base + instructor_group_filter: a group found under that base names the user ({login} and {dn} are substituted, filter-escaped), by default a posixGroup of gid 2000 listing the login in memberUid (the group search that needs no overlay; a groupOfNames site writes "(&(cn=teachers)(member={dn}))"); setting group_search_base enables the test.

In LDAP mode the portal runs them when the user logs in (the role is kept in the session cookie: a change applies at the next login). In proxy mode the reverse proxy authenticates and the tests are run on the request headers (instructor_groups on the groups header, instructor_values on instructor_header: any attribute mod_authnz_ldap exports) and, with ldap_roles = true, by the portal itself with the [auth.ldap] settings for the login the proxy gives (the service account or an anonymous search finds the entry, then the same three tests; the entry’s attr_group is read at the same time, so with that key set the lookup runs for the users the headers already made instructors too, who keep the role whatever the directory says), the result being cached per login for ldap_roles_ttl; a directory failure makes the login a student without group for a minute (logged once) rather than the site unavailable. ldap_groups = true does the group part alone: the directory gives the group of a user whose header carries none, the roles stay those of the headers (the production case: gidNumber exported by Apache, departmentNumber read by the portal for the owners of the Qt GUI projects on the Watch page).

What an instructor gets:

• Watch (the link in the header, /watch): every running project of every reachable host, one row per project with its host, owner (the login), name (the full name of the answers metadata, or the directory’s for a project of the Qt GUI when the portal reads it, above), the owner’s student group when the site configures one (attr_group / header_group), lab, start time, the last action of its owner (how long ago the student last did something on the project: a key pressed, a click or a turn of the wheel anywhere on their project page, which reports them on its events socket at most every 10 s — a tab change, a question picked, the course read, the virtual desktop —, and what the agent sees by itself, to the second: the input of a terminal, an answer, an evaluation, a state; a mouse that only moves is no action. The owner’s actions only: what an instructor helping on the project types and what the spectators of a presentation do never count, so the column tells whether the student is at work. The agent keeps the time across its restarts; the cell is blank for a project the student uses from the Qt GUI only, and for a host whose agent is older than the column), the age of its last archived evaluation, its grade and mark (exo-eval scope, as sre watch shows them), its errors and warnings, the number of self-evaluations the student has run (the agent counts the lines of the project’s .private/auto_eval.log, one per sre eval --auto-eval of the student: live, whereas the auto_eval_count of an archive, the AUTO-EVAL column of sre watch and the Self-evaluations column of the Archives page, is the count before that evaluation), its open terminals and whether the web or the Qt GUI runs it; sortable by any column, filtered by text (host, owner, group, lab) and, with a group source, by the select listing the groups by kind (the departments of the rows, the directory groups and the custom groups of the Groups page, User groups above; every group, one group, or No group: the projects whose owner has no department, those started on the Qt GUI carrying none unless the portal reads the directory, above), the chosen group kept in the URL (?group=kind:name, a bookmark per class group) and remembered by the browser (localStorage): the page comes back with the group chosen last, through the Watch link of the header or in a new tab, until another group or All groups is chosen, a link with ?group= winning and becoming the remembered one; refreshed every 5 s; a row whose last evaluation had errors is red, one without an evaluation younger than 15 minutes (or twice the lab’s evaluation interval) is amber. Open goes to the project, Close stops it. The agents read the archives of their own host (params.archive_dirs) on demand, nothing is scanned while nobody watches. As in sre watch, the page also shows the details: a click on a row opens a dialog with the last archived evaluation of the project (the owner, the lab and the host in the title, the date of the evaluation, the table of its grade elements with the parts, the subtotals, the bonuses, the total and the mark, then the errors and the warnings of the run; Open the project goes to its page; Esc or Close goes back), and a click on the lab cell of a row opens the summary of that lab for the rows shown (the text filter and the group select apply, so a class group alone when wanted; the lab is the file really run, so the projects started through a course directory’s symbolic link to a lab and through the lab’s own path are one lab): a matrix with one row per project and one column per grade element (numbered, the element as the tooltip of the number, the parts as a header row; a cell is the grade, or the OK / MEH / FAIL letter, coloured like the evaluation table; Total and Mark last; a project never evaluated says so), the rows Max, Average and Success (how many of the evaluated projects got the full grade) below it, then the table of the elements with their maximum, minimum, average and distribution of the grades (1.5(2) 3(7)), a subtotal row per part, as the lab summary of sre watch. The elements of the projects are matched by their title, whatever the language of each student. A click on the group cell of a row (when it holds a group) opens the same summary for that lab restricted to the whole group: every project of the lab on the pool whose owner has that group, whatever the text filter (the lab cell keeps the rows shown; an empty group cell opens the last evaluation like the other cells). Both dialogs follow the polls: a new evaluation of the project, or of one of the projects of the lab, reloads the content, a project closed is said so. Reading the results of a student this way never counts as their activity (their project’s idle clock is left alone).

• Archives (the link in the header, /archives): the evaluations collected into the database (The archives of the evaluations above; without [access_log.database] the page says so), the closed projects included. The Evaluations view lists them newest first, one row per evaluation with its time, the student (login, name, group when a group source is configured), the lab, the pool host, the start of the project, the grade and the mark (exo-eval scope), the errors / warnings counts, how many self-evaluations the student had run and whether it was an exam; a row whose evaluation had errors is red. The form searches by Student (the beginning of a login, or a part of a name), Group (a group of the table, with suggestions), Lab (a part of its name, with suggestions), Host, a From / To window on the evaluation time and the exam flag, and chooses what to Show: the last evaluation of each project (the default: the final result of every project, among the evaluations matching the window, the host and the exam flag), the last evaluation of each student (one row per student and lab) or every evaluation (the progression of a project). Rows and the Older / Newer buttons page through the result; a click on a login, group, lab or host cell filters on that value; the filters and the view are kept in the URL (a search can be bookmarked or shared; ?open=<id> opens one evaluation: a link to a result). A click on a row opens the evaluation as the Watch page’s dialog does (the student, the lab and the host in the title, the time of the evaluation and of the start of the project, the table of its grade elements with the parts, the subtotals, the bonuses, the total and the mark, laid out by the browser from the stored elements exactly as the agent lays out the evaluation of a running project, then the errors and the warnings of the run). The Summary view (the second button of the toolbar; a lab is required: without one the view lists the labs of the archives, a click on one loads its summary; the search starts on the last evaluation of each student) is the lab summary of the Watch page for the evaluations found: one block per lab matched (the lab file really run, as on the Watch page: the evaluations of the projects started through a symbolic link to a lab are in the lab’s block, and the lab filter matches the path started as well as the file), with the matrix of the students and the grade elements (Total and Mark last, the rows Max / Average / Success below) and the table of the elements with their statistics; a click on a row of the matrix opens the evaluation. The summary is computed on the 500 newest evaluations found at most (the page says so: narrow the filters). Searching the results of a class group is Group + Lab + Summary; the results of one student across the labs, Student alone. Reports. A checkbox at the left of each row selects evaluations (the header checkbox takes the rows shown; the selection is kept while paging and searching, the bar says how many are selected and how many are not on the page), and the bar under the filters downloads a Spreadsheet (what sre sheet gives: per lab, the table of every evaluation, the Questions statistics and the Sessions best grades), the PDF reports (what sre outline gives: a zip holding one PDF per student and lab, <login>__<lab>.pdf, and outline.ods, the summary) or the Summary spreadsheet (what sre outline -o gives: that outline.ods alone, one row per student and lab with the best evaluation, its time and the mark) of the selected evaluations. A student is a login here, whatever the host: the portal places each project on any host of the pool, so the instances of a lab a student opened several times are merged into one row, one PDF and one Sessions row even when they ran on different hosts (the two commands, run by hand on the archives of a host, keep one per login and hostname: the hostname is the student’s workstation there, and tells apart the users of a shared account); the hostname of a merged row, and of the header of its PDF, is the host of the best evaluation, and each history table of the PDF is headed by the start time and the host of its instance. A selected evaluation stands for its whole project: every stored evaluation of the same projects goes into the reports (the history table of a PDF, every row of the spreadsheet, the best evaluation of the summary), as sre sheet archives/ would do on a host; the Options disclosure unticks that (Every evaluation of the selected projects: the ticked rows alone then) and gives the flags of the two commands: One report / row per project instance (--separate-instances: a student who opened a lab twice gets two reports, or two summary rows, instead of one with two history tables), “Bonus” before the label (--bonus-in-prefix), Evaluation history, Remaining time column (exams) and Group by part (the PDF only: the two spreadsheets ignore them). At most 1000 rows may be ticked; the PDF reports and the summary spreadsheet may cover 150 000 evaluations (300 projects of 300 evaluations give their PDFs in about a minute), the spreadsheet 5000 (the page says so: narrow the selection or untick the option). The file is named after the lab when the selection is of one lab (evaluations_dhcp_20261109-1430.ods, reports_dhcp_20261109-1430.zip, summary_dhcp_20261109-1430.ods), is built by one agent of the pool from the stored rows (The archives of the evaluations above) in the language of the page, and is saved by the browser; the buttons read Generating… meanwhile, a few seconds for a class.

• Pool (the link in the header, /pool): every host of the pool, one row per host with its state as sre-web-tool list shows it (the seconds and the failed attempts of a wake), its projects and terminals, its load, who occupies it, its uptime, its flags (reserved for instructors, power, drain, power off sent, restart sent, and where the preload of the Docker images is: preload queued, preloading 2/5 with the image as tooltip, preload failed with the error), the agent’s version and its last error, the line of sre-web-tool status above the table (in red, with the reason, when the state file of the pool cannot be written: a restart of the portal would undo what is changed on the page); the state cell is green (up), amber (waking, occupied, exam, draining), red (off: the agent does not answer) or grey (disabled, muted text); sortable by any column, filtered by host name (one or several words: the hosts whose name contains any of them, a01 a02 h for those two and every h… host), refreshed every 5 s. The buttons of a row are the commands of the tool: Enable (green) / Disable (red), Wake (light green; a host with a mac or a wake_command, while it is off), Drain, Power off (asked to confirm; allowed by the agent’s allow_power_off, refused while the host has a project or a local session), each disabled with the reason as its tooltip when it does not apply. A checkbox per row (the one of the header ticks the hosts shown by the filter) selects hosts for the bar above the table: the count of the selected hosts (and how many the filter hides), the same five buttons and Preload images (the bar only: the selected hosts wait for their turn to pull the Docker images of the labs, see Pre-loading the images above), each acting on every selected host its action applies to, one call per host run together (the tooltip says how many, the button is disabled when none; Power off asks once with the names; the failures come in one message, the selection stays). Adding and removing hosts, and restarting the agents (restart, shown as the flag restart sent while it lasts), stay with sre-web-tool. Every action, of a row or of the bar, is an admin line of the connection log with the instructor’s login.

• Projects stays the instructor’s own projects (Watch holds the others).

• A student’s project: the student named in the project bar after the lab title (full name and login, from the answers metadata; the login alone when the file has no name); every tab, the devices all open (the privileged sre connect), the answers editable (what the instructor types goes into the student’s answers file, whose metadata stays the student’s: the Watch page and the archives still name the student; every save of either page is pushed to the open pages of the project, so the student’s page shows the instructor’s text within a second and the instructor’s page the student’s); the top of the page (the site bar and the project bar) takes the helper colour (teal) on the instructor’s browser only, a reminder that the project is not theirs (the student’s page stays grey; the presentation colour replaces it while the project is presented); the Terminals tab joins the student’s live terminals when it opens (the instructor sees the student’s shell and may type in it, the student sees the same; a closed sub-tab is not joined again), and the selects offer a private terminal (the student does not see it in their list; what the instructor types there stays with them) or a shared one, the Connect buttons a private one in a window; a terminal on a device the student cannot open themself (a closed machine or switch, a hidden machine) is always private, whatever the choice: the shared kind is greyed for it and the agent makes the session private anyway; the Evaluation tab opens on the last archived evaluation of the project (date, errors in the tooltip), shows each new one by itself (the periodic evaluations of the lab, the student’s own, another instructor’s: the agent tells the open pages) and Start evaluation runs a privileged sre eval (no cooldown, nothing counted on the student’s side, the exo-eval marks); the states apply through the privileged path too. All this is for somebody else’s project: on their own, an instructor is the student (see Instructor mode below).

• Focus student on project: the toggle button of a student’s project page (off by default) brings the student to what the instructor is looking at. While it is on, every page of the student that does not show the project goes to it (now, and any page they open later: the student stays on the project until the button is turned off), the top of both pages (the site bar and the project bar) takes the focus colour (a deeper teal than the helper colour, which comes back on the instructor’s page when the mode is off), the student’s page names the instructor following them, and the two pages share the tab shown, the terminal last gone to, sub-tab or floating window (a terminal the student may see: their own sessions and the shared ones, joined by both: a terminal one page shows is joined by the other, as a sub-tab, when it has not got it; a private terminal of the instructor is never shown nor named to the student, whose page keeps its sub-tab while the instructor is on one; a page brought to the Terminals tab by the focus opens no terminal of its own, as it does when its user clicks the tab for the first time) and the question selected, in both directions: whichever page changes one of them, the other follows, the last change winning. The student’s page also shows the instructor’s mouse pointer (an arrow with their name) and what the instructor selects, as the pages of a presentation show the presenter’s: on the same paragraph, table cell or tab button, on the same row and column of a terminal both pages have, whatever the sizes of the two windows (nothing is drawn while the student is on another tab). A selected text is highlighted without touching the student’s own selection; a selection in a terminal becomes the selection of the student’s terminal. One way only: the instructor does not see the student’s mouse. Nothing is sent of a private terminal of the instructor, nor of a place in a text the instructor reads with the instructor fragments shown (the Instructor mode button on the Informations, Questions and Apply Configuration tabs: the student’s page has not got the same text). Unlike a presentation, the student keeps every control: they type in the terminals, change tab, open terminals, answer questions (and the instructor may write in the answers too, both pages showing the same text). The button’s tooltip says whether the student’s page follows. A reloaded instructor’s page goes on focusing; closing it for a minute, closing the project, or turning the button off releases the student (their page stays where it is, grey again). Any instructor may end a focus; a student is in one focus at a time (the newest wins, an older one still running comes back when it ends).

• Instructor mode on the Open page: the instructor’s own project started with sre start --instructor-mode (the instructor texts kept in info.json, every state listed, the states logged): the Instructor mode button of the project page then shows the instructor() texts of the Informations and Questions tabs, every state and the Log tab, as in the Qt GUI. A student’s project holds no instructor text (the CLI strips them), so the button does not appear there. The Evaluation tab of the instructor’s own project is the student’s one, as in the Qt GUI: Start evaluation runs the self-evaluation (sre --user eval --auto-eval, the cooldown of the lab, the letters OK / MEH / FAIL when the lab hides the marks with no_mark_on_self_grade) and the table it gives stays; the last archived evaluation is shown on somebody else’s project only (an archive holds the numbers).

• Debug mode on the Open page: the third of the Project mode radios of the instructors (Standard, Instructor mode, Debug mode). The instructor’s own project is started with sre start --debug-web-project: the debug project of sre start --debug-project (see Lab Authoring) with the lab given by its name, so that it keeps the lab name of the catalog and is grouped with the lab’s other projects on the Watch and Archives pages. While Debug mode is chosen the lab tree lists every lab of the host, the exam-only and draft ones included (GET /api/labs?all=1, as the Exams page does; the selection is kept when the lab is still listed). The project page then shows what the Qt GUI shows for a debug project: every machine, the hidden ones in red, each with a Connect button, the Full Schema tab (the hidden machines drawn while the Instructor mode button is on), every state, the Log tab with the states and the evaluations, the evaluation table with both scopes, no self-evaluation cooldown. POST /api/projects takes instructor_mode or debug_project (both together is accepted); a student sending either gets 403. Presenting a debug project shows its hidden machines to the spectators, who see the presenter’s page.

• Groups (the link in the header, /groups): the user groups of the site by kind (the departments, the groups of the directory, the custom groups), their members, and the editing of the custom groups (User groups above).

• Exams (the link in the header, /exams): the exams of the web site, programmed for an audience of logins and/or groups, followed and adjusted while they run (Exams above).

• The host of a project: the Host select of the Open page (shown when the pool has a host reserved for instructors, instructors_only = true in [[hosts]]) names one of them, with its state and its projects (greyed when off or full), or leaves the automatic placement: the least loaded host the instructor may use, reserved or shared. POST /api/projects takes the optional host; a student sending one gets 403.

• Present: the Present button of a project page (the instructor’s own project, or a student’s) opens the list of the users connected to the site (a text filter, the groups of every kind with the number of their members connected: the departments, the directory groups and the custom groups of the Groups page, Select all): the chosen students, one by one and by group (a group takes its members connected now and those connecting later), see the project read-only on their own browsers (above). The top of the page (the site bar, the project bar and the bar under it) then takes the presentation colour (lavender), a reminder that the students are watching, and that bar tells how many students are in the audience and how many are watching; Edit audience changes the list, Stop presenting ends it (so does closing the project) and the bars are grey again. The spectators follow the instructor’s page: the tab, the terminals the instructor opens (joined read-only: the students see the same shell, cannot type, and do not change its size), the question, the schema options, the Instructor mode texts, the Display tab view-only, the scroll of the tab shown and the pan and zoom of the schema (followed as the instructor scrolls, a few times a second), the instructor’s mouse pointer (an arrow with their name, sent on the presentation socket about 25 times a second while it moves over the panes, the tab bar or a terminal; nothing is shown while it is elsewhere) and the text the instructor selects (in the page, highlighted without touching the student’s own selection; in a terminal, selected in the student’s read-only terminal), and the Evaluation tab shows them the table of the instructor’s page as it is, sent through the portal after each evaluation, in the instructor’s language (the archive of the project holds numbers only, while the instructor’s own self-evaluation may be letters). A reloaded page goes on presenting; a page closed for a minute ends the presentation. A student is in one presentation at a time (the newest that includes them), and falls back to the previous one when it ends.

• Webconf: the button of the site bar (when [bbb] of web.toml names a BigBlueButton server, Web conference above) opens the rooms: start one for chosen students and groups, join a colleague’s as a moderator, change its audience, end it. The students put in a room get the client in a window over their page.

• Talk privately: on a student’s project, next to Focus student on project: the student is invited to a conversation of the two alone, voice and text (a banner, Join / Not now); once they join, both browsers show a private room instead of the class conference, and come back to it when the conversation ends (Private conversations above).

Administrators

An administrator is an instructor who may read the connection log on the History page (the link in the header, /history). Who is one is decided in web.toml, by login or by group, and a user who is not an instructor never gets the rights whatever the lists say:

• [auth] administrators: a list of logins, in every mode;

• in LDAP mode, administrator_groups of [auth.ldap] (DNs, tested like instructor_groups: the memberOf attribute, or the group_filter search) and administrator_group_filter (a group found under group_search_base naming the user, like instructor_group_filter; no default), run at login for the users the instructor tests accepted (the flag is kept in the session cookie, like the role);

• in proxy mode, administrator_groups of [auth.proxy] on the groups header and, with ldap_roles or ldap_groups (a directory is configured either way), the two [auth.ldap] tests above, run by the portal for the instructors nothing else made administrators (cached per login like the roles; the flag is lost while the directory is down, like the role of ldap_roles).

The History page needs [access_log.database] (The connection log above; without it the page says so) and has three views, chosen by the Sessions / Events / Statistics buttons of its toolbar.

The page opens on the Sessions view: the log as one row per session (The connection log above: one login from one client address, from its first request to the logout, the idle timeout or the shutdown): the sessions still open first, in green, from the portal’s memory (listed even without a database), then the closed ones, newest first, each built from its session_close line: login, name, student group (the column is shown when a group source is configured, attr_group or header_group, like the Group column of the Watch page), address, start, end (the last request: an idle session ends there, not at its expiry), duration, number of requests, the labs of the projects used (the running lab names as a tooltip) and what closed it (Logout, Idle, Shutdown; Still open with the number of pages, terminals and desktops connected). A click on a header sorts the rows by that column, a second click reverses the order (the default is the end time, newest first, an open session counting as not ended yet, hence first); the sort applies to the rows shown, the page, not to the whole log: widen the page with Rows or narrow the filters. Only the login and address prefixes and the time window apply; the window bounds the time a session was closed (session_timeout after the last request for an idle one; an open session is listed when it was active in the window); the other filters are hidden. The show cell of a row opens the events view on that session: its login and address, from the minute of its start to the minute after its closing line. The sessions closed before the name, role and group were logged on the closing line show an empty name and group.

The Events view (?view=events in the URL) lists the events newest first, one row per event with its time (the local time of the browser), event, login, client address, host, project, lab, status, duration and the other fields of the line as key=value; a denied or login_failed line, a status of 400 or more and a failed start are red. The headers sort the rows shown the same way (the time, newest first, by default). The form filters by login and address (prefixes: 10.0.1. is a subnet), event (every event name; in proxy mode without login and login_failed, which never occur there), host (exact), project and lab (substrings; the lab is only on the start lines) and a time window.

The Statistics view (?view=statistics) says who connected over a period and for how long. The period is the From / To window of the form: switching to the view with an empty From fills it with the time 24 hours ago, and an empty To is now (the default period is the last 24 hours; the login and address prefixes apply, an address prefix being a room). A session counts when it was active during the period and only its part inside the period is counted; the time of a user is the union of their sessions (two devices at once are not counted twice); the sessions still open count up to their last request, the administrator’s own among them. A user whose role is not instructor is a student (the sessions closed before the role was logged on the closing line count as students). The page shows the totals (users, students, instructors, groups, sessions, client addresses, the total connection time, the average, median and longest connection time of a student, the most users at once and when, the projects started, the projects used, the evaluations asked and the time spent on the project pages), the groups of the students (students, sessions, time, average, projects, evaluations; shown when a group source is configured, like the Group column; a click on a group lists the users of that group alone, a second click every user again), the users (name, group, role, sessions, connected time, time on the project pages, first and last request in the period, addresses, projects, evaluations, requests; sortable by the headers), the labs (one row per lab, every instance merged: projects, users, students, the time spent on the project pages and its average per user, the starts, the evaluations) and the distribution of the students’ connection time (less than 15 minutes, 15 to 30, 30 to 60, 1 to 2 hours, 2 to 4, more). The time on a project page is that of its project_close lines (the page open as long as its events WebSocket was), clipped to the period and merged per user and lab; a page still open is not counted yet, and the pages of the spectators of a presentation never are; the starts are the start lines with result=created, the evaluations the eval lines answered 200; the requests, projects and evaluations of a session are those of the whole session. The portal reads the lines of those four events from the start of the period to two days after its end (a session or a page active in the period but closed later than that is missed), 200 000 of them at most: beyond, the page says the statistics are computed on the newest rows only and asks to narrow the period.

The Group select of the form (shown when a source of user groups is configured, User groups above: the departments, the directory groups and the custom groups, listed by kind as GET /api/groups gives them) restricts the three views to the members of one group, whatever its kind: the portal resolves the group into the logins of its current members and asks the database for their rows alone (one IN list on the indexed login column, so the sessions, the events or the statistics of a class group are one query; the open sessions are filtered the same way). A group nobody belongs to as far as the portal knows (a custom group emptied or deleted since, a department of nobody) gives nothing and the page says so. The department cell of a session filters on that department. The filter is kept in the URL as group=kind:name (a bare name is a department, like the Watch page’s) and logged as group_filter; without any group source the parameter is refused (400).

In the two list views the filters stay in the URL (the sort of the headers does not), so a filtered view reloads and can be sent to a colleague; Older / Newer page through the rows (100, 500 or 1000 at a time, by a cursor on the row ids: a page is stable while new events arrive; each page is sorted on its own) and a click on a login, address or, in the events view, event, host or project cell filters on that value. Nothing is polled: Search asks again. Each query is a history line of the log itself, with the filters asked (view=sessions for the sessions, view=statistics and the period resolved for the statistics).

In proxy mode (the production site behind Apache) the page shows the same sessions: a session starts at the first request carrying the proxy’s identity headers, right after the browser’s password dialog (no login line: Apache authenticates, the portal only sees the result); the Logout button still closes it on the portal before the browser is sent to logout_url, where Apache’s 401 makes it forget the Basic credentials; a tab closed without a logout ends it by the idle timeout like the other modes; the credentials the browser cached make a later return a new session without any dialog; the client address comes from X-Forwarded-For of the trusted_proxies peer (set in every mode: the proxy’s address alone would merge the devices of a user); the authentications Apache refuses never reach the portal (its own log holds them).

The Display tab

A lab machine declared x11_host draws its graphical applications (Wireshark, Firefox…) on the X server of the host: on a workstation, the student’s own desktop. A pool host has no X server, so the agent runs one virtual desktop per project whose lab has such a machine: a TigerVNC Xvnc with an openbox window manager, started as the unprivileged sreweb-x11 user right after sre start, shown in the Display tab by noVNC (make novnc). The containers reach display :N on TCP port 6000+N of the host with the cookie of the project (the core passes them SRE_DISPLAY and SRE_XAUTH_COOKIE, the .bashrc of the images ≥ 1.31 sets DISPLAY from them); the browser reaches the RFB Unix socket of the server through the agent and the portal, as a WebSocket of binary frames. The desktop follows the size of the tab (Resize the desktop; Fit to window scales it instead), the text box of the bar sends text to the X clipboard and, on the owner’s page only, shows what the applications copy (put into the browser’s clipboard too; an instructor helping on the project can send text but receives none: any X client of the containers can set the selection), Reconnect opens the stream again. The number and the cookie of a project never change: a desktop that died, or that was lost when the agent restarted (its X server is a child of the agent), is started again at the next use of the tab, empty. [display] of agent.toml holds the geometry, the range of display numbers and the commands. When no number of the range is free (every one taken by a project, by the lock file of another X server in /tmp or by a listener on its TCP port, which any local user can leave), a project still starts, without a desktop (no Display tab); the agent logs the reason once. Session recording (asciinema) applies as on a workstation, since it happens in sre connect.

• The answers are saved as the user types (debounced), with the identity of the session.

Security

• The agent refuses every request without its bearer token and the identity headers the portal adds (X-SRE-Login, X-SRE-Fullname, X-SRE-Group, …); the browser never talks to it. With mutual TLS (above) only the holder of a client certificate of the CA reaches the token check, the traffic to the agents is encrypted, and allowed_clients refuses every other address with 403. Without TLS the token and the terminals cross the network in clear: keep the agents on a trusted management network or behind a tunnel (the agent warns at startup).

• The session cookie is signed (HMAC-SHA256), HttpOnly, SameSite=Strict, Secure behind HTTPS; it lives 14 days from the sign-in (a language change sets it again with its original date: a session never slides) and carries the role, group and administrator flag of the sign-in (a change in the directory applies at the next sign-in: Logout, or the expiry); the state-changing calls need the X-Requested-With: sre-web header and the WebSockets a matching Origin; the pages carry a Content-Security-Policy (the origin of the BigBlueButton server added to its frame-src when [bbb] is set: the only other origin a page may frame).

• In proxy mode nothing is stored in a cookie: every request carries the identity set by the reverse proxy, accepted from the peers of trusted_proxies only (a request reaching the portal from anywhere else has no identity, headers or not).

• The directory is reached over TLS (ldaps://, or start_tls on ldap://) and the portal verifies the server certificate on every connection: the chain against the system CA store or [auth.ldap] tls_ca, the host of the uri against the names of the certificate; a StartTLS the server declines aborts the login, and a plain ldap:// is refused at startup unless allow_cleartext = true (the student and instructor passwords, in LDAP mode, and the answers the roles and groups are built from would otherwise be exposed to anyone on the path). The ldap3 library validates nothing by itself.

• One account is one user: in LDAP mode the identity carries the login as the directory stores it (the login attribute of the entry found, else the typed login lower-cased), not what the user typed, since the directory matches Bob for bob and binds either; the role tests naming the login (memberUid, which matches case-exactly) use that form too. In proxy mode the proxy must pass that form (AuthLDAPRemoteUserAttribute in the Apache example).

• Rate limits per user (evaluations, starts; per client address in anonymous mode, where a new pseudonym is free) and per client address (logins); the size of the answers is capped; the reports reading thousands of rows (the Statistics view, the summary of a lab on the Archives page) run two at a time, a third waits ten seconds for a slot then gets a 503 to retry. The sign-in page shows its own messages only (/login?error=<code>: password, rate_limited, directory), never a text taken from the URL.

• The answers/ directory of a project is world-writable (the Qt GUI of the student writes there) and the agent, as root, merges the cheat answers sre state leaves in it (answers/cheat.json): that file is opened without following a symbolic link and read only when it is a regular file with one link, owned by root, sre or the reader, of at most cheat_file_max_size bytes (one million); anything else planted under that name is removed (the entry, never what a link points to) and logged once.

• The connection log ([access_log]) keeps who did what from where: the sessions with their duration and projects, the logins and their failures, the denied calls, the administration calls. The only header it reads is X-Forwarded-For, from the peers of trusted_proxies, for the client address; the identity never comes from it. With [access_log.database] the same lines are in a MariaDB / MySQL table, read on the History page by the administrators only ([auth] administrators and the administrator group tests: instructors, never students; every consultation is itself logged); the account of the portal needs SELECT, INSERT, DELETE on that table and nothing else.

• A project’s owner is the third field of its running lab name; the portal, the agent and, in user mode, the sre CLI itself (params.check_project_owner: the local GUI of a pool host lists and reaches the same projects as the CLI lets through) check it. An instructor (see Instructors above) reaches every project, but the agent never runs a command as the student for them: a non-owner’s eval, state and connect are privileged sre calls under the instructor’s own login, the answers a non-owner writes go into the student’s file with the student’s metadata kept (the file never names the instructor), and a private terminal is listed to instructors only; a terminal an instructor opens on a device the student may not open themself (a closed machine or switch, a hidden machine) is private whatever the page asks, since a shared session is joined without the device check. The operations log of an instructor-mode project goes to the subscribers allowed to read it only.

• What the applications of a project’s desktop copy (the X selection, which any X client of the containers can set) is received by the owner’s browser only, where the Display tab puts it into the clipboard; the page of an instructor helping on the project does not listen to it. Likewise the hyperlinks a program prints in a terminal (OSC 8 sequences) open, after a confirmation naming the target, on the owner’s page only; on an instructor’s or a spectator’s page they are inert, the target shown as a tooltip.

• The spectators of a presentation reach the presented project through the portal only, on a short list of read-only calls (the project, its events, the schema, the log, a terminal to join, the display; the evaluation table comes from the presenter’s page through the portal, never from the agent), relayed to the agent under the presenter’s identity (the agent trusts the portal for every identity anyway); everything else is refused, the portal never forwards what a spectator’s browser sends on a terminal socket and the agent attaches a read-only client that cannot type nor resize, and the display socket of a spectator goes through a filter dropping the RFB input and resize messages (an unexpected message closes it). From the browsers, the presentation socket takes the pointer frames of the presenter’s page only (a JSON object of at most 1 KiB locating the mouse, at most 60 a second, every other frame ignored; a spectator’s frames are ignored); the pointer and the selection name nodes of the page and cells of a terminal, never a pixel, and a spectator’s page resolves them on its own layout. The presentations live in the portal’s memory.

The virtual desktops. Each Xvnc listens on TCP port 6000+N on every interface of the pool host, protected by the MIT cookie of its project only, exactly like the X server of a workstation configured for the labs (installation.md); an nftables rule such as nft add rule inet filter input tcp dport 6010-6099 iifname != "docker0" drop keeps the other hosts out. The RFB socket of a desktop is reachable by root only (the agent); the window manager runs as sreweb-x11 with no menu, no launcher and no key binding running a program of the host; the browser reaches a desktop through the owner check of the portal and of the agent.

Limitations

• After an update of the tree, the portal and the agents must be restarted for the Python code (sre-web-tool restart --all for the agents, see Restarting the agents above; the templates are re-read when they change); the scripts and style sheets of web/static are served with Cache-Control: no-cache, so a browser revalidates them on every page load (an ETag answers 304 for an unchanged file) and never runs a script older than the page. A page loaded before the update keeps its scripts until it is reloaded.

• The virtual X desktops die with the agent (a restart of its unit): they are started again, empty, at the next use of the Display tab; the applications shown must be launched again.

• The preloads of the Docker images are counted per portal (two portals sharing hosts each give max_preloads turns), need the portal (an agent never preloads by itself), are asked at the start of an agent or by hand only (no periodic check: a cron job running sre-web-tool preload --all does it) and are not kept across a restart of an agent: a preload asked by hand and not started yet is forgotten, one in progress is stopped ([preload] at_start asks again). A host that cannot be polled while it preloads stops counting, so one more host may get a turn meanwhile.

• The ports of the bridged machines are bound on the pool host and shown with its name.

• The exam mode of the Qt GUI (exam.json on a host) still excludes that host from the pool: the web site has its own exams (Exams above), per student rather than per host. The save of the terminal sessions across a restart of the agent is a later phase.

• A terminal is lost when the agent restarts; its sub-tab offers a Reconnect button. A page that left keeps its terminals for terminals_linger only (120 minutes by default); where each one was (a sub-tab, a window) is remembered by the browser, another browser of the same user gets them back as sub-tabs.

• The replay a joining viewer gets may start inside an escape sequence; the next prompt or redraw of the application settles the display.

• The [limits] eval_interval of the portal (10 s between two evaluations of a login) also applies to an instructor evaluating several students’ projects in a row.

• A presentation ends when the portal restarts (nothing is kept on disk); the agents log the spectators under the presenter’s login (the connection log of the portal has the real ones); a spectator’s terminal shows the presenter’s grid at a smaller font when their window is smaller; the presenter’s pointer and selection are shown where the student’s page holds the same element: a student reading the lab in another language sees the pointer on the matching paragraph and no selection there; in a terminal they follow the text (the rows are counted from the cursor, a selection is found again by the text of its first and last rows), so a selection of rows older than the student’s 64 KiB replay is not shown, the student’s terminal does not follow the scrollback position of the presenter’s (a pointer on rows the student’s screen does not show is hidden), and when several users type in the presented terminal its pty takes the smallest window, so a wider presenter’s terminal wraps the long lines elsewhere and the pointer and the selection on them may miss; the pointer is not shown while the presenter’s page has no socket open; a presented project never idles out while it is watched. A focus ends when the portal restarts too; while one is on, the student’s other browser tabs all go to the project.

• A web conference is forgotten when the portal restarts (the meeting goes on on the BigBlueButton server until it expires); every page of the site joins the meeting again (a few seconds); two tabs of one user hand the client over to each other, but two pages loading at the same instant may both join.

Development

make web-tests runs the offline tests of the web front-end: the agent against a fake sre (tests/mock_sre.py), the portal against that agent, the templates. python3 -m sreweb.agent --config agent.toml --listen 127.0.0.1:8451 and python3 -m sreweb.portal --config web.toml --listen 127.0.0.1:8450 --web-dir web (from src/, in the venv) run both locally; with sre_exe pointed at a wrapper of tests/mock_sre.py (SRE_PUB_DIR exported in the wrapper), the whole site works without Docker. The French catalog is locale/fr/LC_MESSAGES/sreweb.po (hand-maintained, make web-translations).

The pure JavaScript modules (the lab tree of the Open page, the group keys, the exam form, the floating terminal windows) are tested under node or bun; on a machine with neither, make tests runs them under bun in a throwaway container of the oven/bun image instead (SRE_TEST_JS=docker, what the Makefile passes then: the image is pulled once, the tests skip without a reachable Docker daemon, root or the docker group; SRE_TEST_JS=docker:<image> names another image, SRE_TEST_JS= leaves them skipped). Installing a runtime (apt install nodejs) is the lighter alternative.

Stress test

scripts/web-stress.py simulates a class of students on a portal through its HTTP / WebSocket API, as the browser drives it: for each student (logins stress001, stress002…), the Projects page (GET /api/projects every 5 s, the presence socket), the Open page (GET /api/labs), POST /api/projects read as NDJSON until the result line (a 503 or 429 is retried after its retry_after), the project page (the project JSON, the schema, the events socket kept open and reconnected with the browser’s backoff, POST .../touch every 60 s), a terminal on the first device open to the student with a few commands typed (the echo of the first one is checked), then, over --hold seconds, --answers saves and --evals self-evaluations, and finally the stop. The students begin spread over --ramp seconds. --instructor adds an instructor on the Watch page (GET /api/watch, the Pool page’s calls, every 5 s), --admin-token samples GET /api/pool every 5 s (the reachable flag and the project count of each host as the portal sees them), --probe measures GET /api/me every 2 s. The tool needs Python 3.11 and aiohttp only (the venv of the installation has them) and runs in proxy authentication mode: the identity of each student travels in the X-Remote-* headers, so it must run on a host of trusted_proxies (the portal’s own host, against http://127.0.0.1:8450); the instructor is made by X-Remote-Gid: 2000 (--instructor-header / --instructor-value otherwise).

The summary on stdout gives, per phase (start, project_get, events_ws, terminal_ws, eval, stop…), the number of calls, the errors, the latencies (p50 / p95 / max) and the statuses, then the students completed, the checks failed (own_projects_only: a listing showed another student’s project; project_listed: the new project was missing from GET /api/projects; terminal_echo; eval_grades; events_stayed_open; no_unexpected_removed; stopped), the hosts used and the pool samples (the flips of reachable show the status polls timing out). --report FILE writes everything as JSON (one entry per student), --events FILE a CSV timeline (one line per request or socket event). The exit status is 0 when every student completed with every check true. --keep leaves the projects running; --cleanup stops every project of the synthetic logins (through the instructor identity) and exits.

Stage 1: a fake pool, no Docker

scripts/web-fake-pool.py launches from the checkout N real agents whose sre is a wrapper of tests/mock_sre.py (the projects are directories, the terminals /bin/sh, a start takes 7 × --start-delay seconds, an evaluation --eval-delay), and a real portal in proxy mode trusting the loopback, with the production [limits] (--start-interval / --eval-interval override them), max_projects per host and an admin token; then it runs the command given after -- with SRE_STRESS_PORTAL, SRE_STRESS_ADMIN_TOKEN and SRE_STRESS_WORKDIR in its environment (the defaults of --portal and --admin-token) and stops everything. Every MOCK_* variable of its environment reaches the fake sre (MOCK_NO_MARK=1: the letters of a self-evaluation, MOCK_EVAL_ERRORS, MOCK_COOLDOWN, see the header of tests/mock_sre.py):

venv/bin/python3 scripts/web-fake-pool.py --hosts 4 --max-projects 25 -- \
    venv/bin/python3 scripts/web-stress.py --users 100 --ramp 60 --hold 120 --instructor --probe \
        --report report.json --events events.csv

Without a command the pool stays up until Ctrl-C (portal=... admin_token=... workdir=... is printed first), for runs by hand. The work directory (configs, portal.log, access.log, one agent.log and sre.log per host) is removed at the end unless --keep-workdir or a daemon died; the end of the log of a daemon that died is printed with the failure. The agents listen on the ports after the portal’s (--listen, 18450 by default): keep them below the ephemeral ports of the system (32768 on Linux), where a port may be held by a client connection, even for a minute after it closed. --administrators prof,... makes those logins administrators of the portal ([auth] administrators), so that an instructor among them (X-Remote-Gid: 2000) can exercise the History page; its Sessions view lists the open sessions without any database.

Stage 2: the production pool

On the portal’s host, outside class hours, with the tool copied there (or /opt/sre/scripts/ once synced) and the venv of the installation:

1. Preflight. The portal must see the tool as a trusted peer: from the host itself when the portal container uses the host network (docker inspect sre-web --format '{{.HostConfig.NetworkMode}}'), from inside the container otherwise (docker exec). Read the admin token in [admin] of web.toml. sre-web-tool status and list: the hosts up and the spare slots; with fewer slots than students, the pool wakes hosts during the ramp (wake_timeout, the students refused with 503 “a host is starting” retry after the retry_after of the answer, 30 s). Check ulimit -n of the portal process (two relayed sockets per student). Note the archive_dirs of the hosts: every evaluation writes an archive there.

2. Smoke run: web-stress.py --portal http://127.0.0.1:8450 --users 3 --hold 60 --lab sre/dhcp.py --instructor --admin-token ... --probe must give 3 students completed and no check failed.

3. Full run: the same with --users 100 --ramp 300 --hold 600 --report ... --events .... Meanwhile: docker stats sre-web (the CPU of one event loop, the memory), the connection log (event=start ... result=, denied, hosts unreachable), sre-web-tool list (states, projects per host), the journal of an agent (journalctl -u sre-agent -f), the probe’s p95 in the summary.

4. Cleanup: web-stress.py --cleanup ... stops what a stopped run left (a Ctrl-C stops the students’ projects by itself), then remove the archives of the synthetic logins (*@@@stress0* under archive_dirs on the hosts) and check sre-web-tool list is back to 0 projects.

The reverse proxy and the directory are bypassed (the tool speaks to the portal), and follow_local_sessions never applies (the synthetic logins have no local session).