HDDS-14900. Document S3 Lifecycle Configurations - Object Expiration#9979
HDDS-14900. Document S3 Lifecycle Configurations - Object Expiration#9979xichen01 wants to merge 2 commits intoapache:HDDS-8342from
Conversation
|
|
||
| | Property | Description | | ||
| |----------|-------------| | ||
| | `ozone.lifecycle.service.enabled` | Whether to enable the lifecycle management service. | |
There was a problem hiding this comment.
Let's show the default value of each property too.
| - In OM HA mode, only the leader OM executes lifecycle evaluation tasks. | ||
| - For FSO buckets, a directory is only marked as expired and deleted if all its child files and subdirectories have expired. | ||
| - For FSO buckets using Prefix, if the Prefix does not end with `/`, it will match both the directory with the exact name and sibling directories starting with the same prefix (e.g., `dir` matches both `dir` and `dir1`). | ||
| - If an invalid lifecycle configuration exists in the OM RocksDB, the service will skip that configuration and log an error without affecting the processing of other buckets. |
There was a problem hiding this comment.
With the HDDS-13548 fix, I believe that we don't have any invalid lifecycle configuration exists in OM DB, right?
There was a problem hiding this comment.
Yes, this is just a theoretical possibility, but it might not need to be included in the user documentation.
Let me delete this
|
|
||
| 1. The lifecycle evaluation tasks running on the old leader will be interrupted. | ||
| 2. After the new leader is elected, the lifecycle service restarts from the beginning. Previously evaluated buckets are not skipped, and the task starts over from the first bucket. | ||
| 3. If `ozone.lifecycle.service.run.interval` is set to a large value (e.g., the default `24h`), the next round of tasks may not be scheduled until the following day after a leader transfer. |
There was a problem hiding this comment.
The BackgroundService schedules task with 0s delay, so after leader transfer, lifecycle service of new leader will start task immediately.
There was a problem hiding this comment.
Oh, Yes, that's a feature in our internal version; it's not the community feature. I will delete this
| In an OM HA deployment, the lifecycle service only runs on the leader OM. When a Transfer Leader operation is performed: | ||
|
|
||
| 1. The lifecycle evaluation tasks running on the old leader will be interrupted. | ||
| 2. After the new leader is elected, the lifecycle service restarts from the beginning. Previously evaluated buckets are not skipped, and the task starts over from the first bucket. |
There was a problem hiding this comment.
This reminds one optimization we can do, that is, introduce a finish timestamp of bucket scan in LifeCycleConfiguration. If OM is restarted or OM leader is transferred, we can skip scan these LifeCycleConfiguration if (now() - it's last finish timestamp) < scan interval. What do you think?
There was a problem hiding this comment.
We can add a lastScanningKey field for each rule in the DB, the new leader can list bucket from the lastScanningKey, this is how we implement Transition Action internally.
There was a problem hiding this comment.
We can consider add both, lastVerifyTimestamp and lastScanningKey.
|
|
||
| The compaction operation is executed asynchronously. Check the corresponding OM node's logs for completion status. | ||
|
|
||
| ## References |
There was a problem hiding this comment.
Can we add a section for the supported metrics ahead of Reference.
There was a problem hiding this comment.
I'm concerned that changes to the metrics might not be synchronized to the document.
In the future, we might be able to export all metrics to the document using a plugin.
|
Thanks @xichen01 , the doc overall looks good to me. |
|
Hi @xichen01 is this a design doc or user documentation? User docs should go to the new website in the apache/ozone-site repo. Design docs are still being committed to this repo though. |
@errose28 , this is a user doc. But since the code is still in feature branch, not merged back to master yet, I think maybe it's stay with ozone repo first. After the feature branch merge, we can submit a new patch for this for ozone-site repo, what do you think? BTW, there is no auto sync between ozone repo and ozone-site repo for document now? |
A better option might be to raise a draft PR in ozone-site and merge it when the feature branch is merged here. The formatting supported for pages is a little different, so it is not always a copy/paste from one to the other. Docusaurus has callouts and mdx that could be helpful, for example. ozone-site also has stricter linting rules.
For user docs we should probably delete docs and Hugo content from the Ozone repo to ensure all new user docs go to ozone-site. We have the previous docs captured as Hugo builds in the new website so removing them here would ensure they live on as read-only. The only sync process we may want to add would be for design docs. |
3 participants
What changes were proposed in this pull request?
Document S3 Lifecycle Configurations - Object Expiration
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-14900
How was this patch tested?
--