Re: [PATCH v2 5/6] automation/qtb: add QTB framework README

[Oleksii Kurochko <oleksii.kurochko@xxxxxxxxx>]

On 8/27/26 11:42 AM, Baptiste Le Duc wrote:
> Document the qtb riscv64 test framework in a README.
I understand that you are trying to introduce things so they could be 
traceble. But probaly it makes mentioned in cover letter or somewhere 
else that thre is README which probably should be read first for easier 
understanding of the patch series.
> It covers:
>    - the core concepts (machine, test type, test) and how they map to files
>    - the source files layout
>    - the CLI: `qemu_smoke_riscv64.py <type> <command>`, with "console-test"
>      as the type
>    - the config files: the machine catalog and a type's own `<type>.yaml`
>    - the Jinja2 device-tree templates under dts/
>    - how to add a test (config-only) and how to add a new test type.
>
> Assisted-by: Claude:claude-opus-5
> Signed-off-by: Baptiste Le Duc <baptiste.le-duc@xxxxxxxxxx>
> ---
>   automation/scripts/qtb/riscv/README.md | 182 +++++++++++++++++++++++++
>   1 file changed, 182 insertions(+)
>   create mode 100644 automation/scripts/qtb/riscv/README.md
>
> diff --git a/automation/scripts/qtb/riscv/README.md 
> b/automation/scripts/qtb/riscv/README.md
> new file mode 100644
> index 0000000000..c173714d75
> --- /dev/null
> +++ b/automation/scripts/qtb/riscv/README.md
> @@ -0,0 +1,182 @@
> +qtb riscv64 test framework
> +==========================
> +
> +A small framework that boots Xen under QEMU on riscv64 and drives it to a
> +pass/fail verdict automatically from its console output.
> +
> +It is built on QEMU's qtb (QEMU Test Bench) Python package, which gives
> +programmatic control of a QEMU process over QMP and qtest, plus access to the
> +consoles.
> +
> +What it does
> +------------
> +
> +1. Reads a machine description (what to boot: cpus, Xen command line) and a
> +   test description (what to assert).
> +2. Generates the host device tree.
> +3. Launches QEMU with Xen.
> +4. Reads the console and checks what Xen printed.
> +
> +Core concepts
> +-------------
> +
> +- Machine: test-agnostic description of what to boot. Reusable across test
> +  types. `config.yaml` -> `MachineConfig`.
> +- Test type: a `RiscvQtbTest` subclass implementing the logic of a kind of test
> +  (e.g. `console-test`). Identified by `type_id`. See `console_test/`.
> +- Test: one named, runnable instance of a type: a machine plus the type's
> +  parameters. Lives in the type's `<type>.yaml`.
> +
> +A test type owns a config file describing its tests, each test names a machine
> +from the shared catalog (`config.yaml`) and layers its own parameters on top.
> +
> +Layout
> +------
> +
> +```
> +qemu_smoke_riscv64.py        CLI entry point (<type> run | list)
> +
> +qtb/riscv/                   This framework
> +  __init__.py                Public API
> +  qtb_test.py                RiscvQtbTest ABC every test type derives from
> +  config.py                  Machine catalog parser -> MachineConfig
> +  xen_dt.py                  Generates the host device tree from its Jinja2 
> template
> +  dt.py                      Compile .dts -> .dtb with dtc
> +  paths.py                   Path resolution (pkg-relative)
> +  machine.py                 RiscvTestMachine: assembles the QEMU command line
> +
> +  config.yaml                The machine catalog (shared across test types)
> +  dts/                       Jinja2 device-tree templates (host, common)
There is only one template, qemu-host.dts.j2, nothing "common".

> +
> +  console_test/              The console-test type
> +    __init__.py
> +    console_test.py          ConsoleTest implementation
> +    console-test.yaml        Its tests
> +
> +  unit/                      pytest unit tests of the framework logic itself
> +```
> +
> +How a type is selected
> +----------------------
> +
> +The test type is the first positional argument (`qemu_smoke_riscv64.py 
> console-test run
> +...`). The CLI builds one subcommand per entry of `TEST_TYPES` (`__init__.py`),
> +named after the type's `type_id`.
> +
> +Each type declares the `config_file` it reads its tests from.
> +
> +Prerequisites
> +-------------
> +
> +- `qemu.qtb`, QEMU's Python package (`python/` in the QEMU tree)
qemu.qtb isn't in QEMU's python/ (it isn't upstream yet, as the cover
letter says), so this sends people to the wrong place. Please point to
the fork and the commit used by the container in patch 1, or better,
say that the simplest way to get all prerequisites is that container,
and show how to run a test in it with containerize.

> +- `jinja2`, `pyyaml`, `pexpect`
> +- `dtc` (device-tree-compiler)
> +- `qemu-system-riscv64`
> +- the `xen` binary a machine boots
> +
> +CLI usage
> +---------
> +
> +Run from `automation/scripts/`, or give the full path from the Xen tree root
> +(`./automation/scripts/qemu_smoke_riscv64.py ...`), which is what CI does.
Neither works without QTB_BINARIES_DIR unless xen is copied to
automation/scripts/qtb/riscv/binaries/ (see my reply to patch 3), and
the README doesn't say where to put it.

> +
> +```
> +# List every test the type defines in its config:
> +./qemu_smoke_riscv64.py console-test list
> +
> +# Run one test (drives it to PASS/FAIL, exit 0/1):
> +./qemu_smoke_riscv64.py console-test run 
> dom0less-1smp-0domu-1vcpu-aplic-imsic-null \
> +    --log-dir qtb-logs
> +```
> +
> +`--log-dir` (run only) collects the QEMU process log, the qtest log, and each
> +console as `con<N>.log`: `con0.log` is Xen's own console, the only one wired
There is no qtest log: QtbMachine passes "-qtest-log none". The
riscv64-qtest-<pid>.log file is QEMU's stdout/stderr; "qtest" comes
from the machine name.

> +up today. Omit it to write no logs. `-v/--verbose` raises the
> +log level to debug.
> +
> +Config files
> +------------
> +
> +`config.yaml` is the machine catalog. `binaries:` are build artifacts resolved
> +under `binaries/` (overridable with `$QTB_BINARIES_DIR`); absolute paths pass
> +through.
This reads as binaries/ at the top of the tree, while it is actually
automation/scripts/qtb/riscv/binaries/.

> +
> +Machine entries omit any optional field left at its default.
> +Here are the parameters:
> +
> +- `pcpu` (required): host physical cpus.
> +- `mmu_type` (default `sv48`): Xen host MMU type.
See my reply to patch 2: Xen doesn't use this.

> +- `xen_bootargs` (default `""`): Xen command line.
> +
> +`<type>.yaml` describes the tests of that type.
> +
> +`console-test`
> +--------------
> +
> +A test names a machine and maps a console index to the string(s) expected on
> +that console: index 0 is Xen's own console (`con0`, logged as `con0.log`).
> +
> +```
> +machine_catalog: config.yaml    # the catalog to resolve machine names against
> +tests:
> +  dom0less-1smp-0domu-1vcpu-aplic-imsic-null:
> +    machine: dom0less-1smp-0domu-1vcpu-aplic-imsic-null   # a name in 
> config.yaml
> +    expect:
> +      0: [All set up]           # Xen itself must print "All set up"
> +```
> +
> +Logic, per console:
> +
> +1. read the console
> +2. wait for each expected string in turn, in the order listed
> +3. bound each wait by `timeout` seconds, retrying a timed-out wait up to
> +   `attempts` times
See my reply to patch 3: nothing is retried, attempts only multiplies
the timeout.

[...]

~ Oleksii