Skip to main content

Import From Raw Logs

This page shows a recommended raw-log structure that can be converted into a Bublik-importable run bundle.

Example Log​

Use these downloadable example log and script to convert it:

The raw log shows the recommended structure for:

  • nested packages;
  • logical tests and iterations;
  • scoped parameter blocks;
  • summary lines with expected/verdict annotations;
  • measurement and artifact-style lines.

Conventions (what the converter should rely on)​

  • Timestamped message lines: YYYY-MM-DD HH:MM:SS,mmm:[LEVEL][ENTITY] message...
  • Multiline blocks: continuation lines are indented (example: Run metadata:, PackageParameters:, TestParameters:, IterationParameters:).
  • Explicit hierarchy boundaries:
    • PackageStart: <name> / PackageEnd: <name>
    • TestStart: <name> / TestEnd: <name> status=<STATUS>
    • IterationStart: <name> iter=<n> / IterationEnd: <name> iter=<n> status=<STATUS>
  • Stable names: use the same canonical package and test names in start/end markers and summary lines so the adapter can map children reliably.
  • Parameters: keep parameter blocks scoped to the current open node:
    • PackageParameters: for the current package.
    • TestParameters: for the logical test shared by all its iterations.
    • IterationParameters: for one concrete test execution.
  • Measurements: emit one or more MEASUREMENT: key=value ... lines; these become Bublik measurements/charts.
  • Artifacts: emit plain artifact or measurement-summary lines such as WRITE: io=158GiB (170GB) when you want the adapter to derive artifacts from raw output. Use explicit ARTIFACT: name=relative/path lines for direct file attachments.
  • Scenario-friendly steps: emit distinguishable lines such as Step: ... so the adapter can map them into scenario rows in the UI.
note

This page defines a recommended raw-log structure.
The actual conversion step is responsible for mapping this structure into Bublik’s import bundle format.

  • PackageStart creates a pkg node in bublik.json and pushes it onto the current hierarchy stack.
  • Nested packages (package -> package -> package) become nested pkg nodes in both plan and iters.
  • TestStart identifies the logical test under the current package path.
  • IterationStart identifies one executable leaf node under that logical test.
  • If one logical test has multiple iterations, prefer a parent grouping node for the logical test and one leaf test node per iteration.
  • PackageParameters: map to the current package node params.
  • TestParameters: map to the logical test node params.
  • IterationParameters: map to the leaf iteration node params.
  • Run identity and filtering fields such as PROJECT, CFG, START_TIMESTAMP, and RUN_STATUS belong in meta_data.json, not in bublik.json node params.

#Scenario / #Test readiness​

To make #Scenario and #Test work correctly in the log UI, raw logs should provide enough structure for the adapter to derive:

  • the main test entity for each test page;
  • scenario-action rows (Step, StepPush, StepPop);
  • result rows (Verdict, Artifact, Self, Run) tied to the correct entity.

Raw logs do not need to contain the literal UI marker TAPI Jumps, but they do need explicit test boundaries and stable test entity names so the adapter can emit that row correctly in json/node_*.json.

See UI log JSON for the UI-side mapping rules.

What Bublik needs to import (bundle requirements)​

Bublik imports a bundle directory available over HTTP.

Bundle layout (example)​

${BUBLIK_DOCKER_DATA_DIR}/logs/logs/.../bundle-dir
|-- bublik.json
|-- meta_data.json
|-- json
|   |-- tree.json
|   |-- node_1_0.json
|   |-- node_id1.json
|   |-- node_id2.json
|   |-- ...
|   `-- node_id{N}.json
`-- {source}.log

bublik.json​

  • Purpose: main run import file — execution plan, iteration tree, package/test nodes, timestamps, obtained and expected results, verdicts, errors, artifacts, measurements, tags, and display paths.
  • Database import: required.
  • Log UI: required indirectly (drives node JSON generation).
  • Details: see bublik.json.

meta_data.json​

  • Purpose: run metadata — test suite, configuration, device, start/finish timestamps, project, run status, optional extra metas.
  • Database import: required.
  • Log UI: used for run identity and filters.
  • Details: see meta_data.json.

json/tree.json​

  • Purpose: navigation index — maps node JSON filenames to a tree structure for the log UI (main_package plus per-file entries and optional children).
  • Database import: not required.
  • Log UI: required for navigation as produced by this bundle format.
  • Details: see UI log JSON.

json/node_1_0.json​

  • Purpose: root package log-viewer JSON for the full run.
  • Database import: not required.
  • Log UI: required for the full run log view.
  • Details: see UI log JSON.

json/node_id1.json​

  • Purpose: same content as node_1_0.json — compatibility with UI/proxy paths that address the root node by test id (test_id 1).
  • Database import: not required.
  • Log UI: recommended.
  • Details: see UI log JSON.

json/node_id{N}.json​

  • Purpose: per-iteration log pages for packages and tests.
  • Database import: not required.
  • Log UI: required for individual package/test log pages.
  • Details: see UI log JSON.

{source}.log​

  • Purpose: copy of the original input for traceability and parser debugging.
  • Database import: not used directly.
  • Log UI: optional (not loaded as structured log JSON).

Summary: database import needs bublik.json and meta_data.json. Opening logs in the UI needs the json/ directory.

Import flow (diagram)​

note

If this diagram renders as a code block in your docs site, enable Mermaid in the documentation build (Docusaurus needs Mermaid support turned on).

Step-by-step import​

  1. Convert raw log → bundle directory
    • Produce the bundle files listed above.
    • Example:
python3 example_bundle.py \
  example_raw.log \
  -o /tmp/example-bundle \
  --project example \
  --pretty
  1. Transfer bundle directory to the Bublik host
    • If you generated the bundle on a different machine, copy the whole bundle directory to the host where Bublik is running.
    • Example:
scp -r /path/to/bundle-dir user@<bublik_host>:/tmp/bundle-dir
  1. Publish the bundle directory
    • Place the bundle directory anywhere under ${BUBLIK_DOCKER_DATA_DIR}/logs/logs/ (you can add any nesting and use any folder names).
    • Confirm it is accessible as a directory URL under http://<bublik_host>/logs/.../ (you should see meta_data.json, bublik.json, and json/).
    • Example on the Bublik host:
mkdir -p ${BUBLIK_DOCKER_DATA_DIR}/logs/logs/example
cp -R /tmp/example-bundle ${BUBLIK_DOCKER_DATA_DIR}/logs/logs/example/
  1. Import into Bublik
curl 'http://<bublik_host>/api/v2/importruns/source/?url=http://<bublik_host>/logs/example/example-bundle&force=true'

The script writes:

  • bublik.json
  • meta_data.json
  • json/tree.json
  • json/node_1_0.json
  • json/node_id1.json
  • json/node_id<N>.json
  • a copy of the source log

Quick checklist​

  • meta_data.json has PROJECT, RUN_STATUS, and stable run key metas (typically START_TIMESTAMP + CFG).
  • Raw logs use explicit package/test/iteration start and end markers, so the adapter can reconstruct nested children reliably.
  • Package-, test-, and iteration-scoped parameters are logged separately and mapped to the correct params node.
  • Every test iteration has a deterministic name/path and a single final status (PASSED/FAILED/INCOMPLETE/...).
  • Measurements appear under the same test node they were produced by (not at run level), otherwise charts may not show up where expected.
  • Raw logs contain enough structure for the adapter to derive the main test entity and scenario rows for #Scenario / #Test.