On 14/08/2018 22:17, Federico Razzoli wrote:
> Hi,
>
> Just a small correction: ZoL does not support O_DIRECT, but FreeBSD ZFS
> does. Probably other distributions also do.
>
> Regards,
> Federico Razzoli

Hi, I just tried on FreeBSD 11.x a small C program with O_DIRECT support
[1] and it really seems O_DIRECT is ignored: writes go into ARC and are
served from it when data is read. ZFS compression for the dataset it off.

This do not surprise me: O_DIRECT implies zero-memory-copy and/or DMA
from main memory to the disk themselves. While with standard filesystem
this should be possible, with CoW+checksum (and anything which
transforms data when they flow, ie: compression) this become very difficult.

Back to main point... anyone with some insights on doublewrite and ZFS?

# Before running the test program:
ARC Size:                              0.09%  1.14    MiB
        Target Size: (Adaptive)        100.00% 1.20    GiB
        Min Size (Hard Limit):          12.50%  153.30  MiB
        Max Size (High Water):          8:1    1.20    GiB

# After running it:
ARC Size:                              48.65%  596.61  MiB
        Target Size: (Adaptive)        100.00% 1.20    GiB
        Min Size (Hard Limit):          12.50%  153.30  MiB
        Max Size (High Water):          8:1    1.20    GiB

# Reading the just-written file shows data are server by ARC (ie: too
fast for coming from the disk)
root@freebsd:~ # dd if=/tank/test.img of=/dev/null bs=1M
512+0 records in
512+0 records out
536870912 bytes transferred in 0.188852 secs (2842809718 bytes/sec)


[1] Test program:
root@freebsd:~ # cat test.c
#define _GNU_SOURCE
#include <string.h>
#include <stdlib.h>
#include <fcntl.h>
#include <stdio.h>
#include <unistd.h>
#define BLOCKSIZE 128*1024

int main()
{
        void *buffer;
        int i = 0;
        int w = 0;
        buffer = malloc(BLOCKSIZE);
        buffer = memset(buffer, 48, BLOCKSIZE);
        int f = open("/tank/test.img", O_CREAT|O_TRUNC|O_WRONLY|O_DIRECT);
        for (i=0; i<512*8; i++) {
                w = write(f, buffer, BLOCKSIZE);
        }
        close(f);
        free(buffer);
        return 0;