HDDS-14888. Improve DiskCheckUtil.checkReadWrite to tolerate disk full

[ChenSammi]

What changes were proposed in this pull request?

DiskCheckUtil.checkReadWrite does file write and read check to make sure disk is writable and readable.
If disk is full, it will catch the exception and return false.

The outer StorageVolume#check will decide whether mark this time volume check as fail or not, depending on whether volume has more than 100B *2 free space.
As the disk space will dynamically change, so there is change that checkReadWrite() is failed due to disk full, while when below logic is evaluated, it has enough available space again.
if (!diskChecksPassed && getCurrentUsage().getAvailable() < minimumDiskSpace) {

The goal of this target is to analysis the exception in checkReadWrite() directly, so that disk full case will not return false.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-14888

How was this patch tested?

new unit test

The exception when test with local disk full volume is, FileSystemException for both FileChannel open and output stream write.

java.nio.file.FileSystemException: /Volumes/DiskFullTest/disk-check-ae1d2075-200a-409f-aeb4-655471324a4d: No space left on device
  at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
  at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
  at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
  at java.base/sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:182)
  at java.base/java.nio.channels.FileChannel.open(FileChannel.java:292)
  at java.base/java.nio.channels.FileChannel.open(FileChannel.java:345)
  at org.apache.ratis.util.FileUtils.lambda$newFileChannel$6(FileUtils.java:178)
  at org.apache.ratis.util.LogUtils.supplyAndLog(LogUtils.java:58)
  at org.apache.ratis.util.FileUtils.newFileChannel(FileUtils.java:177)
  at org.apache.ratis.util.FileUtils.newOutputStreamForceAtClose(FileUtils.java:165)
  at org.apache.ratis.util.FileUtils.newOutputStreamForceAtClose(FileUtils.java:169)

[sumitagrawl]

@ChenSammi Thanks for working over this, please check message with OS dependent case

[sumitagrawl]

message used to check the failure is OS dependent. We need check if message is same in all supported OS variant or not, docker, mac

[ashishkumar50]

Below is just safe guard check, may be we can keep. We are depending on exception message which may differ in different OS.