Skip to content

[DRAFT] Update benchmarks#1014

Draft
justsmth wants to merge 9 commits intoaws:mainfrom
justsmth:update-benchmarks
Draft

[DRAFT] Update benchmarks#1014
justsmth wants to merge 9 commits intoaws:mainfrom
justsmth:update-benchmarks

Conversation

@justsmth
Copy link
Copy Markdown
Contributor

@justsmth justsmth commented Jan 22, 2026

Description of changes:

Adds a new aws-lc-rs-bench crate — a standalone benchmarking tool that measures CPU instruction counts using Valgrind's callgrind. Unlike wall-clock benchmarks, instruction counts are deterministic and noise-free, which makes them well-suited for catching unexpected performance changes in CI.

The tool covers AEAD, digest, HMAC, HKDF, key agreement, and signature operations (55 benchmarks total). A GitHub Actions workflow runs these on every PR and pushes a comparison report as a PR comment.

Also cleans up aws-lc-rs-testing:

  • Makes ring and openssl truly optional (previously they were unconditional dev-deps behind empty feature flags)
  • Gates the cross-library comparison tests with #![cfg(feature = "...")]
  • Modernizes the RSA verify benchmarks to use UnparsedPublicKey instead of the deprecated VerificationAlgorithm trait
  • Inlines the agree_ephemeral calls in agreement benchmarks to accommodate the differing aws-lc-rs / ring signatures

Call-outs:

  • On this PR the "Benchmarks" CI jobs will fail b/c there is nothing to compare them to.
  • aws-lc-rs-bench is Linux-only for instruction counting (requires Valgrind). A walltime subcommand is available for quick local testing on any platform.
  • The crabgrind crate is pulled in only on Linux via cfg(target_os = "linux").
  • The aws-lc-rs-testing dependency changes mean cargo test -p aws-lc-rs-testing (without features) now compiles the ring/openssl comparison tests to zero test cases. Pass --features ring-benchmarks,openssl-benchmarks to run them.

Testing:

  • cargo check -p aws-lc-rs-bench and cargo test -p aws-lc-rs-testing --features ring-benchmarks,openssl-benchmarks both pass.
  • Verified walltime mode runs all 55 benchmarks end-to-end.
  • Verified that without features, all aws-lc-rs-testing test targets correctly compile to 0 tests (feature gates work).

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and the ISC license.

@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented Jan 22, 2026

Codecov Report

❌ Patch coverage is 50.00000% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.61%. Comparing base (c358484) to head (a1b8e0d).
⚠️ Report is 351 commits behind head on main.

Files with missing lines Patch % Lines
aws-lc-rs/src/cipher.rs 50.00% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1014      +/-   ##
==========================================
- Coverage   95.80%   92.61%   -3.19%     
==========================================
  Files          61       71      +10     
  Lines        8143    10051    +1908     
  Branches        0    10051   +10051     
==========================================
+ Hits         7801     9309    +1508     
- Misses        342      450     +108     
- Partials        0      292     +292     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@justsmth justsmth force-pushed the update-benchmarks branch 11 times, most recently from a9596b5 to 9786ef1 Compare January 22, 2026 21:01
@justsmth justsmth force-pushed the update-benchmarks branch 3 times, most recently from 2095a6a to 7b488af Compare March 3, 2026 15:43
@justsmth justsmth force-pushed the update-benchmarks branch from 7b488af to 51a1a26 Compare March 3, 2026 17:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants