ZFS resilver from 4m/s to 284m/s

3个3t磁盘做成的raidz，更换一块磁盘，resilver速度只有4m/左右,需要200多个小时才能完成，难以接受，使用iostat监控磁盘性能，新加入的sdd竟然100% busy，从网上搜索到一些文章，请见参考附件。优化过程如下：

1 设置zfs参数

[root@ZFS ~]# cat /etc/modprobe.d/zfs.conf 
options zfs zfs_arc_max=2147483

> Blockquote

648 zfs_top_maxinflight=64 zfs_resilver_min_time_ms=5000 zfs_resilver_delay=0

优化后速度无提升。

2 开启硬盘write cache

hdparm -W1 /dev/sdb

优化后速度提升到20m/s

3 调整硬盘调度策略

echo deadline > /sys/block/sda/queue/scheduler

默认调度策略为cfq,可以尝试以下deadline和noop, 我测试下来发现noop最快。优化后显示resilver速度达到250m以上，现在的zfs status -v 结果如下:

  pool: ny
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Fri Jan 22 00:45:59 2016
    1.90T scanned out of 3.22T at 303M/s, 1h16m to go
    631G resilvered, 58.95% done
config:

        NAME                 STATE     READ WRITE CKSUM
        ny                   DEGRADED     0     0     0
          raidz1-0           DEGRADED     0     0     0
            sdb1             ONLINE       0     0     0
            sdc1             ONLINE       0     0     0
            replacing-2      DEGRADED     0     0     0
              /x/nydisk.img  OFFLINE      0     0     0
              sdd2           ONLINE       0     0     0  (resilvering)

errors: No known data errors

iostat显示的磁盘读写速度在150m左右

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.00    0.00    6.24    0.00    0.00   93.76

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdc               0.00     0.00 2553.00    0.00   167.99     0.00   134.76     1.22    0.48    0.48    0.00   0.29  73.60
sdb               0.00     0.00 2577.00    0.00   167.84     0.00   133.39     0.90    0.34    0.34    0.00   0.22  55.60
sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdd               0.00     0.00    0.00 1387.00     0.00   169.96   250.96     0.89    0.64    0.00    0.64   0.64  89.20

这个瞬时io速度变化比较大。

[1] http://www.matt-j.co.uk/2014/06/25/zfs-on-linux-resilver-scrub-performance-tuning/
[2] http://list.zfsonlinux.org/pipermail/zfs-discuss/2014-October/019076.html