Conversation
github-actions
bot
added
A-docs
This adds a test framework to test the code snippets in the guide (the docs/ directory). Currently, the framework defines four different test configurations: * `toml`, tested by parsing as a Cot config file * `html.j2`, tested by trying to build an Askama template * `rust`, tested by building the snippet as an extension of a default project template (as generated by `cot-cli`) * `rust,has_main`, a version of the above that also defines its own `main` function It should be fairly easy to extend this by defining more configurations, if needed. The framework defines a number of utilities to make the testing more pleasant: * Many symbols are imported by default * Similarly to rustdoc, lines can be prepended with `# ` to indicate lines that are not displayed in the guide, but required for the test to build * For the Askama tests, there are some dummy files and structures created This is part of the major effort to make the guide always up-to-date. As a result of this, a number of code snippets were modified as they did not build before. This concludes the work on #471.
There was a problem hiding this comment.
Pull request overview
Adds an integration-test framework to compile/check code blocks in the docs/ guide, and updates multiple guide snippets so they build under the new framework.
Changes:
- Introduces
cot-testcrate +libtest-mimicrunner to validate fenced code blocks indocs/*.md(Rust/TOML/Askama templates). - Updates many guide snippets (hidden
# ...lines, async fixes, etc.) to compile under the new checking approach. - Adds supporting glue:
just test-docs, nextest concurrency override, and small framework/API touch-ups (e.g.,TestRequestBuilder::cache,ForeignKey::new).
Reviewed changes
Copilot reviewed 25 out of 27 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| justfile | Adds test-docs task/alias to run the new doc-snippet tests. |
| docs/upgrade-guide.md | Adjusts a Rust code fence and list indentation. |
| docs/testing.md | Updates examples; adds async fixes; minor formatting changes. |
| docs/templates.md | Updates Askama examples and adds extra supporting code for compilation. |
| docs/static-files.md | Adds hidden lines so static-file snippets compile. |
| docs/site/Cargo.lock | Bumps cot-site git rev in the docs site lockfile. |
| docs/README.md | Reworks headings and adds placeholder section for doc-test instructions. |
| docs/openapi.md | Updates OpenAPI examples (return types, derives, route registration) to compile. |
| docs/introduction.md | Adds hidden scaffolding + a rust,has_main snippet for the main example. |
| docs/forms.md | Updates form examples to use newer APIs (into_response()?) and adds hidden scaffolding. |
| docs/db-models.md | Makes DB examples compile (adds hidden model scaffolding; comments out sample URLs). |
| docs/caching.md | Adds hidden scaffolding so cache examples compile. |
| docs/admin-panel.md | Updates admin-panel snippets for newer APIs and adds missing scaffolding. |
| cot/src/test.rs | Adds TestRequestBuilder::cache() behind cache feature. |
| cot/src/project.rs | Updates a panic message to reference cot::Project. |
| cot/src/db/relations.rs | Adds ForeignKey::new(...) convenience constructor. |
| cot-test/tests/doc_code_blocks.rs | New libtest-mimic runner that parses Markdown and runs per-block checks. |
| cot-test/templates/html_main.rs | Template-compilation harness for Askama snippets. |
| cot-test/templates/base.html | Base template used by Askama snippet compilation. |
| cot-test/src/lib.rs | New helper crate that generates a temp project and runs cargo check for snippets. |
| cot-test/Cargo.toml | Defines new internal crate and custom test harness entry. |
| cot-macros/tests/ui/func_query_invalid_field.stderr | Updates expected UI test stderr output wording. |
| cot-macros/tests/ui/attr_main_args.stderr | Updates expected stderr to reference cot::Project. |
| cot-macros/tests/ui/attr_main_args.rs | Updates return type to cot::Project. |
| Cargo.toml | Adds cot-test to workspace + new workspace deps (comrak, libtest-mimic, etc.). |
| Cargo.lock | Updates lockfile for new deps and cot-test crate. |
| .config/nextest.toml | Adds nextest grouping to serialize doc-code-block tests. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| let final_code = match test_type { | ||
| "has_main" => code.to_string(), | ||
| _ => Self::wrap_in_main(&preamble, code), | ||
| }; |
There was a problem hiding this comment.
has_main snippets skip the generated preamble/common imports entirely (final_code is just code.to_string()). This makes rust,has_main blocks depend on having all imports/allow attributes written explicitly, and currently breaks at least the docs/introduction.md snippet that uses impl Project without importing Project. Consider still prepending the preamble (and the #![allow(...)] attributes) for has_main, while only skipping the wrap_in_main wrapper so snippets can define their own main.
| ```rust,has_main | ||
| # struct CotTutorialProject; | ||
| # impl Project for CotTutorialProject {} | ||
| #[cot::main] | ||
| fn main() -> impl Project { | ||
| CotTutorialProject |
There was a problem hiding this comment.
This rust,has_main snippet uses impl Project but doesn't import/qualify Project (e.g., use cot::Project; or -> impl cot::Project). As written, it won't compile standalone, and it will also fail the doc-code-block tests unless the harness injects the common imports for has_main blocks.
| ## Testing the code | ||
|
|
||
| TODO |
There was a problem hiding this comment.
The new "Testing the code" section is left as TODO. Since this PR introduces a doc-snippet test framework (cot-test) and a just test-docs target, it would be better to document the exact command(s) to run and any prerequisites (e.g., just test-docs / cargo nextest run -p cot-test).
| "examples/sessions", | ||
| "examples/todo-list", | ||
| "examples/forms", | ||
| "examples/forms", "cot-test", |
There was a problem hiding this comment.
Workspace member list formatting: "examples/forms", "cot-test" is on a single line while the rest of the members entries are one-per-line. Keeping one entry per line makes future diffs cleaner and matches the surrounding style.
| "examples/forms", "cot-test", | |
| "examples/forms", | |
| "cot-test", |
| The end-to-end testing framework consists of two parts: the [`cot::e2e_test`](attr@cot::e2e_test) macro and the [`TestServerBuilder`](struct@cot::test::TestServerBuilder) struct. The [`cot::e2e_test`](attr@cot::e2e_test) macro allows you to define end-to-end tests that allow you to run your project in the background, while the [`TestServerBuilder`](struct@cot::test::TestServerBuilder) struct allows you to create a test server that you can send your requests to. An example of how to use the [ | ||
| `cot::e2e_test`](attr@cot::e2e_test) macro and the [`TestServerBuilder`](struct@cot::test::TestServerBuilder) struct is shown below in a simple test that checks if the server is running and returns the `Hello world!` response: |
There was a problem hiding this comment.
Markdown link markup is broken by the line break after the [. This will likely render incorrectly (or not as a link) in the generated guide. Consider keeping the [...] link text on the same line (or removing the stray [ entirely).
| The end-to-end testing framework consists of two parts: the [`cot::e2e_test`](attr@cot::e2e_test) macro and the [`TestServerBuilder`](struct@cot::test::TestServerBuilder) struct. The [`cot::e2e_test`](attr@cot::e2e_test) macro allows you to define end-to-end tests that allow you to run your project in the background, while the [`TestServerBuilder`](struct@cot::test::TestServerBuilder) struct allows you to create a test server that you can send your requests to. An example of how to use the [ | |
| `cot::e2e_test`](attr@cot::e2e_test) macro and the [`TestServerBuilder`](struct@cot::test::TestServerBuilder) struct is shown below in a simple test that checks if the server is running and returns the `Hello world!` response: | |
| The end-to-end testing framework consists of two parts: the [`cot::e2e_test`](attr@cot::e2e_test) macro and the [`TestServerBuilder`](struct@cot::test::TestServerBuilder) struct. The [`cot::e2e_test`](attr@cot::e2e_test) macro allows you to define end-to-end tests that allow you to run your project in the background, while the [`TestServerBuilder`](struct@cot::test::TestServerBuilder) struct allows you to create a test server that you can send your requests to. An example of how to use the [`cot::e2e_test`](attr@cot::e2e_test) macro and the [`TestServerBuilder`](struct@cot::test::TestServerBuilder) struct is shown below in a simple test that checks if the server is running and returns the `Hello world!` response: |
| ```rust | ||
| use std::fmt::Display; | ||
| use cot::Template; | ||
|
|
||
| struct Item { | ||
| title: String, | ||
| } | ||
|
|
||
| # struct Item { title: String } | ||
| # #[derive(Template)] | ||
| # #[template(path = "index.html")] | ||
| # struct IndexTemplate { item: Item } | ||
| impl Display for Item { | ||
| fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { | ||
| write!(f, "{}", self.title) | ||
| } | ||
| } |
There was a problem hiding this comment.
This snippet uses impl Display for Item but Display isn't imported anywhere in this page anymore. To keep the example copy/pasteable, either qualify it (impl std::fmt::Display for Item) or add a (possibly hidden) use std::fmt::Display; line.
|
|
||
| # struct Item; | ||
| # impl std::fmt::Display for Item { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { Ok(()) } } | ||
| impl HtmlSafe for Item {} |
There was a problem hiding this comment.
This HtmlSafe example no longer imports askama::filters::HtmlSafe, so impl HtmlSafe for Item {} won't compile as-is. Consider qualifying the trait (askama::filters::HtmlSafe) or adding back a (possibly hidden) use askama::filters::HtmlSafe; line.
| impl HtmlSafe for Item {} | |
| impl askama::filters::HtmlSafe for Item {} |
|
Bencher Report| Branch | docs-tests-2 |
| Testbed | github-ubuntu-latest |
Click to view all benchmark results
| Benchmark | Latency | Benchmark Result milliseconds (ms) (Result Δ%) | Upper Boundary milliseconds (ms) (Limit %) |
|---|---|---|---|
| empty_router/empty_router | 📈 view plot 🚷 view threshold | 6.72 ms(+10.72%)Baseline: 6.07 ms | 7.37 ms (91.20%) |
| json_api/json_api | 📈 view plot 🚷 view threshold | 1.21 ms(+14.71%)Baseline: 1.06 ms | 1.26 ms (96.48%) |
| nested_routers/nested_routers | 📈 view plot 🚷 view threshold | 1.08 ms(+11.40%)Baseline: 0.97 ms | 1.15 ms (94.50%) |
| single_root_route/single_root_route | 📈 view plot 🚷 view threshold | 1.06 ms(+13.70%)Baseline: 0.94 ms | 1.11 ms (95.63%) |
| single_root_route_burst/single_root_route_burst | 📈 view plot 🚷 view threshold | 17.24 ms(-2.45%)Baseline: 17.68 ms | 21.08 ms (81.80%) |
This adds a test framework to test the code snippets in the guide (the docs/ directory). Currently, the framework defines four different test configurations:
toml, tested by parsing as a Cot config filehtml.j2, tested by trying to build an Askama templaterust, tested by building the snippet as an extension of a default project template (as generated bycot-cli)rust,has_main, a version of the above that also defines its ownmainfunctionIt should be fairly easy to extend this by defining more configurations, if needed.
The framework defines a number of utilities to make the testing more pleasant:
#to indicate lines that are not displayed in the guide, but required for the test to buildThis is part of the major effort to make the guide always up-to-date. As a result of this, a number of code snippets were modified as they did not build before.
This concludes the work on #471.
Type of change