Lima の vmType VZ と virtiofs を試す

Limaversion 0.14.0 で QEMU だけではなく macOS の Virtualization.Framework に対応していました。

vmtype という設定項目が増えています。

この新しい Framework では Host のディレクトリをマウントするのに virtiofs が使えるようになっており、QEMU での reverse-sshfs9p よりもパフォーマンスが良いらしいので試してみます。

とりあえず fio を使ってみますが、Host (mac) でのキャッシュの状況の影響を受けるだろうし、ディレクトリのリストやファイルの open、close が遅い場合は測れないけどとりあえず参考までにということで。で、試した結果 fio だけではイマイチだったので tarball の展開にかかる時間も比較してみた。

lima は普段 docker 用の VM として使っているので docker の volume としてマウントして docker コンテナ内で fio コマンドを実行しています。

$ limactl --version
limactl version 0.14.2

vz (Virtualization.Framework) は macOS 13 以降でしか使えないみたいです。また、ARM の Guest として Intel VM を実行することもできないみたいです。

$ sw_vers
ProductName:            macOS
ProductVersion:         13.1
BuildVersion:           22C65
# fio --version
fio-3.25

VM 作成

lima の VM は examples/docker.yaml をベースにして vmTypemountType を変更して作成しました。

$ limactl ls
NAME                STATUS     SSH                VMTYPE    ARCH      CPUS    MEMORY    DISK      DIR
qemu-9p-docker      Stopped    127.0.0.1:0        qemu      x86_64    2       2GiB      100GiB    ~/.lima/qemu-9p-docker
qemu-ssh-docker     Stopped    127.0.0.1:0        qemu      x86_64    2       2GiB      100GiB    ~/.lima/qemu-ssh-docker
vz-virtfs-docker    Running    127.0.0.1:52261    vz        x86_64    2       2GiB      100GiB    ~/.lima/vz-virtfs-docker

VM の情報はこんな感じ。

$ uname -r
5.15.0-56-generic
$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.1 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

docker context

以前 blog (Docker on Lima) を書いたときは DOCKER_HOST という環境変数を指定することで VM 上の docker daemon にアクセスするようにしていましたが。limactl start で VM を作成した時に表示される メッセージ が更新されていて docker context というものを知りました。今後はこれを使うことにします。

$ docker context ls
NAME                      DESCRIPTION                               DOCKER ENDPOINT                                                 KUBERNETES ENDPOINT   ORCHESTRATOR
default                   Current DOCKER_HOST based configuration   unix:///var/run/docker.sock                                                           swarm
desktop-linux                                                       unix:///Users/teraoka/.docker/run/docker.sock
lima-qemu-9p-docker                                                 unix:///Users/teraoka/.lima/qemu-9p-docker/sock/docker.sock
lima-qemu-ssh-docker                                                unix:///Users/teraoka/.lima/qemu-ssh-docker/sock/docker.sock
lima-vz-virtfs-docker *                                             unix:///Users/teraoka/.lima/vz-virtfs-docker/sock/docker.sock
rancher-desktop           Rancher Desktop moby context              unix:///Users/teraoka/.rd/docker.sock

fio で IOPS を比較

次のコマンドを使って fio を実行しました。

docker pull debian:11.6
docker run --rm -it -v $(pwd):/work debian:11.6 bash
apt-get update && apt-get install -y fio bc
fio /work/test.fio --section=random-read
fio /work/test.fio --section=random-write
fio /work/test.fio --section=random-readwrite

test.fio

test.fio ファイルの中身は次のもの。

[global]
size=2*1024*1024*$mb_memory
directory=/work
bs=4k
numjobs=$ncpus
thread=1
runtime=60s
direct=1
ioengine=libaio
group_reporting=1

[random-read]
rw=randread

[random-write]
rw=randwrite

[random-readwrite]
rw=randrw
rwmixread=70
rwmixwrite=30

IOPS 結果

各環境で5回実行。うーん、IOPS だけだと 9p と virtiofs との差はなんとも言えない感じ。

vmTypemountTypeRandom Read IOPSRandom Write IOPSRandom Read/Write IOPS
QEMUreverse-sshfs1,4952,7551,037 / 443
QEMUreverse-sshfs1,7726041,172 / 503
QEMUreverse-sshfs1,7023,2791,135 / 487
QEMUreverse-sshfs1,7642,9191,153 / 495
QEMUreverse-sshfs1,7711,9391,139 / 488
QEMU9p5,8144,8394,030 / 1,729
QEMU9p5,7034,1433,819 / 1,637
QEMU9p8,9164,4933,797 / 1,628
QEMU9p9,0374,9723,923 / 1,682
QEMU9p8,9105,0263,965 / 1,699
VZvirtiofs9,6452,2654,207 / 1,805
VZvirtiofs7,7164,4813,874 / 1,662
VZvirtiofs7,7404,5753,761 / 1,614
VZvirtiofs7,7474,5423,806 / 1,633
VZvirtiofs7,5654,3143,681 / 1,579

QEMU reverse-sshfs

Random Read
# fio /work/test.fio --section=random-read
random-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
...
fio-3.25
Starting 2 threads
Jobs: 2 (f=2): [r(2)][100.0%][r=6058KiB/s][r=1514 IOPS][eta 00m:00s]
random-read: (groupid=0, jobs=2): err= 0: pid=3036: Fri Dec 30 10:58:22 2022
  read: IOPS=1495, BW=5982KiB/s (6126kB/s)(351MiB/60002msec)
    slat (usec): min=9, max=1322, avg=50.91, stdev=41.56
    clat (usec): min=79, max=9807, avg=1222.00, stdev=225.85
     lat (usec): min=683, max=9884, avg=1287.87, stdev=235.52
    clat percentiles (usec):
     |  1.00th=[  824],  5.00th=[  898], 10.00th=[  963], 20.00th=[ 1057],
     | 30.00th=[ 1106], 40.00th=[ 1156], 50.00th=[ 1188], 60.00th=[ 1237],
     | 70.00th=[ 1303], 80.00th=[ 1385], 90.00th=[ 1500], 95.00th=[ 1614],
     | 99.00th=[ 1893], 99.50th=[ 2008], 99.90th=[ 2343], 99.95th=[ 2442],
     | 99.99th=[ 3654]
   bw (  KiB/s): min= 5420, max= 6417, per=100.00%, avg=5987.08, stdev=93.17, samples=238
   iops        : min= 1354, max= 1604, avg=1496.26, stdev=23.36, samples=238
  lat (usec)   : 100=0.01%, 500=0.01%, 750=0.11%, 1000=13.10%
  lat (msec)   : 2=86.26%, 4=0.52%, 10=0.01%
  cpu          : usr=0.22%, sys=11.38%, ctx=90655, majf=0, minf=4
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=89735,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=5982KiB/s (6126kB/s), 5982KiB/s-5982KiB/s (6126kB/s-6126kB/s), io=351MiB (368MB), run=60002-60002msec
Random Write
# fio /work/test.fio --section=random-write
random-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
...
fio-3.25
Starting 2 threads
random-write: Laying out IO file (1 file / 3938MiB)
Jobs: 2 (f=2): [f(2)][100.0%][eta 00m:00s]
random-write: (groupid=0, jobs=2): err= 0: pid=3045: Fri Dec 30 11:00:06 2022
  write: IOPS=2755, BW=10.8MiB/s (11.3MB/s)(647MiB/60059msec); 0 zone resets
    slat (usec): min=9, max=5484, avg=157.21, stdev=164.50
    clat (usec): min=9, max=1689.0k, avg=480.19, stdev=11320.34
     lat (usec): min=34, max=1689.2k, avg=658.14, stdev=11319.37
    clat percentiles (usec):
     |  1.00th=[    10],  5.00th=[    11], 10.00th=[    11], 20.00th=[    11],
     | 30.00th=[    11], 40.00th=[    12], 50.00th=[    13], 60.00th=[    18],
     | 70.00th=[    26], 80.00th=[   149], 90.00th=[   494], 95.00th=[   668],
     | 99.00th=[  1004], 99.50th=[  1303], 99.90th=[185598], 99.95th=[198181],
     | 99.99th=[219153]
   bw (  KiB/s): min=   70, max=24816, per=100.00%, avg=11491.88, stdev=5356.66, samples=230
   iops        : min=   16, max= 6204, avg=2872.40, stdev=1339.28, samples=230
  lat (usec)   : 10=1.46%, 20=64.61%, 50=11.52%, 100=1.24%, 250=3.77%
  lat (usec)   : 500=7.60%, 750=6.62%, 1000=2.17%
  lat (msec)   : 2=0.74%, 4=0.09%, 10=0.02%, 100=0.01%, 250=0.16%
  lat (msec)   : 500=0.01%, 750=0.01%, 2000=0.01%
  cpu          : usr=0.25%, sys=18.02%, ctx=197836, majf=0, minf=2
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,165512,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=10.8MiB/s (11.3MB/s), 10.8MiB/s-10.8MiB/s (11.3MB/s-11.3MB/s), io=647MiB (678MB), run=60059-60059msec
Random Read/Write
# fio /work/test.fio --section=random-readwrite
random-readwrite: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
...
fio-3.25
Starting 2 threads
Jobs: 2 (f=2): [m(2)][100.0%][r=4180KiB/s,w=1889KiB/s][r=1045,w=472 IOPS][eta 00m:00s]
random-readwrite: (groupid=0, jobs=2): err= 0: pid=3026: Fri Dec 30 10:54:45 2022
  read: IOPS=1037, BW=4150KiB/s (4249kB/s)(243MiB/60002msec)
    slat (usec): min=9, max=16301, avg=424.26, stdev=590.42
    clat (usec): min=23, max=14736, avg=1301.33, stdev=288.54
     lat (usec): min=728, max=18995, avg=1740.40, stdev=662.20
    clat percentiles (usec):
     |  1.00th=[  840],  5.00th=[  947], 10.00th=[ 1012], 20.00th=[ 1090],
     | 30.00th=[ 1156], 40.00th=[ 1205], 50.00th=[ 1254], 60.00th=[ 1319],
     | 70.00th=[ 1401], 80.00th=[ 1483], 90.00th=[ 1631], 95.00th=[ 1778],
     | 99.00th=[ 2147], 99.50th=[ 2343], 99.90th=[ 2966], 99.95th=[ 3490],
     | 99.99th=[ 6521]
   bw (  KiB/s): min= 3776, max= 4499, per=100.00%, avg=4152.97, stdev=65.06, samples=238
   iops        : min=  944, max= 1124, avg=1037.43, stdev=16.19, samples=238
  write: IOPS=443, BW=1775KiB/s (1818kB/s)(104MiB/60002msec); 0 zone resets
    slat (usec): min=9, max=7192, avg=76.19, stdev=98.69
    clat (usec): min=10, max=5483, avg=142.69, stdev=167.88
     lat (usec): min=37, max=7217, avg=236.52, stdev=184.30
    clat percentiles (usec):
     |  1.00th=[   12],  5.00th=[   13], 10.00th=[   13], 20.00th=[   33],
     | 30.00th=[   41], 40.00th=[   68], 50.00th=[  105], 60.00th=[  133],
     | 70.00th=[  161], 80.00th=[  200], 90.00th=[  310], 95.00th=[  461],
     | 99.00th=[  824], 99.50th=[  979], 99.90th=[ 1336], 99.95th=[ 1483],
     | 99.99th=[ 2376]
   bw (  KiB/s): min= 1412, max= 2258, per=100.00%, avg=1776.47, stdev=81.07, samples=238
   iops        : min=  352, max=  564, avg=443.55, stdev=20.30, samples=238
  lat (usec)   : 20=4.74%, 50=5.62%, 100=4.13%, 250=11.00%, 500=3.19%
  lat (usec)   : 750=0.97%, 1000=6.62%
  lat (msec)   : 2=62.46%, 4=1.24%, 10=0.02%, 20=0.01%
  cpu          : usr=0.15%, sys=15.19%, ctx=105253, majf=0, minf=2
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=62246,26629,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=4150KiB/s (4249kB/s), 4150KiB/s-4150KiB/s (4249kB/s-4249kB/s), io=243MiB (255MB), run=60002-60002msec
  WRITE: bw=1775KiB/s (1818kB/s), 1775KiB/s-1775KiB/s (1818kB/s-1818kB/s), io=104MiB (109MB), run=60002-60002msec

QEMU 9p

Random Read
# fio /work/test.fio --section=random-read
random-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
...
fio-3.25
Starting 2 threads
Jobs: 2 (f=2): [r(2)][100.0%][r=23.9MiB/s][r=6130 IOPS][eta 00m:00s]
random-read: (groupid=0, jobs=2): err= 0: pid=3106: Fri Dec 30 13:48:01 2022
  read: IOPS=8910, BW=34.8MiB/s (36.5MB/s)(2088MiB/60001msec)
    slat (usec): min=56, max=5699, avg=220.50, stdev=115.35
    clat (nsec): min=1191, max=629060, avg=2086.80, stdev=3940.19
     lat (usec): min=58, max=5701, avg=223.01, stdev=115.55
    clat percentiles (nsec):
     |  1.00th=[  1416],  5.00th=[  1688], 10.00th=[  1720], 20.00th=[  1752],
     | 30.00th=[  1784], 40.00th=[  1832], 50.00th=[  1880], 60.00th=[  1912],
     | 70.00th=[  1944], 80.00th=[  1976], 90.00th=[  2008], 95.00th=[  2064],
     | 99.00th=[  2320], 99.50th=[  5024], 99.90th=[ 66048], 99.95th=[ 70144],
     | 99.99th=[117248]
   bw (  KiB/s): min=21208, max=60848, per=100.00%, avg=35763.03, stdev=8202.08, samples=238
   iops        : min= 5302, max=15212, avg=8940.66, stdev=2050.51, samples=238
  lat (usec)   : 2=87.13%, 4=12.34%, 10=0.07%, 20=0.12%, 50=0.05%
  lat (usec)   : 100=0.27%, 250=0.02%, 500=0.01%, 750=0.01%
  cpu          : usr=2.83%, sys=15.14%, ctx=531748, majf=0, minf=4
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=534646,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=34.8MiB/s (36.5MB/s), 34.8MiB/s-34.8MiB/s (36.5MB/s-36.5MB/s), io=2088MiB (2190MB), run=60001-60001msec
Random Write
# fio /work/test.fio --section=random-write
random-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
...
fio-3.25
Starting 2 threads
Jobs: 1 (f=1): [_(1),f(1)][100.0%][eta 00m:00s]
random-write: (groupid=0, jobs=2): err= 0: pid=3115: Fri Dec 30 13:49:03 2022
  write: IOPS=5026, BW=19.6MiB/s (20.6MB/s)(1178MiB/60013msec); 0 zone resets
    slat (usec): min=55, max=101622, avg=393.87, stdev=2114.52
    clat (nsec): min=1245, max=306020, avg=2095.78, stdev=3597.79
     lat (usec): min=57, max=101626, avg=396.38, stdev=2114.64
    clat percentiles (nsec):
     |  1.00th=[  1624],  5.00th=[  1672], 10.00th=[  1704], 20.00th=[  1736],
     | 30.00th=[  1768], 40.00th=[  1800], 50.00th=[  1848], 60.00th=[  1912],
     | 70.00th=[  1944], 80.00th=[  1976], 90.00th=[  2040], 95.00th=[  2128],
     | 99.00th=[  3056], 99.50th=[ 11840], 99.90th=[ 63744], 99.95th=[ 70144],
     | 99.99th=[110080]
   bw (  KiB/s): min=10618, max=56916, per=100.00%, avg=20183.88, stdev=5565.14, samples=238
   iops        : min= 2653, max=14229, avg=5045.76, stdev=1391.31, samples=238
  lat (usec)   : 2=83.48%, 4=15.73%, 10=0.21%, 20=0.21%, 50=0.13%
  lat (usec)   : 100=0.23%, 250=0.01%, 500=0.01%
  cpu          : usr=2.94%, sys=7.08%, ctx=300439, majf=0, minf=2
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,301634,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=19.6MiB/s (20.6MB/s), 19.6MiB/s-19.6MiB/s (20.6MB/s-20.6MB/s), io=1178MiB (1235MB), run=60013-60013msec
Random Read/Write
# fio /work/test.fio --section=random-readwrite
random-readwrite: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
...
fio-3.25
Starting 2 threads
Jobs: 1 (f=1): [f(1),_(1)][100.0%][r=2201KiB/s,w=859KiB/s][r=550,w=214 IOPS][eta 00m:00s]
random-readwrite: (groupid=0, jobs=2): err= 0: pid=3124: Fri Dec 30 13:50:05 2022
  read: IOPS=3965, BW=15.5MiB/s (16.2MB/s)(929MiB/60001msec)
    slat (usec): min=158, max=4769, avg=332.10, stdev=71.38
    clat (nsec): min=1235, max=513370, avg=2157.14, stdev=4381.52
     lat (usec): min=160, max=4777, avg=334.68, stdev=71.56
    clat percentiles (nsec):
     |  1.00th=[  1592],  5.00th=[  1672], 10.00th=[  1704], 20.00th=[  1736],
     | 30.00th=[  1784], 40.00th=[  1832], 50.00th=[  1880], 60.00th=[  1912],
     | 70.00th=[  1944], 80.00th=[  1976], 90.00th=[  2024], 95.00th=[  2064],
     | 99.00th=[  3024], 99.50th=[ 15936], 99.90th=[ 68096], 99.95th=[ 74240],
     | 99.99th=[130560]
   bw (  KiB/s): min= 8040, max=21888, per=99.97%, avg=15858.13, stdev=2840.22, samples=238
   iops        : min= 2010, max= 5472, avg=3964.50, stdev=710.07, samples=238
  write: IOPS=1699, BW=6799KiB/s (6962kB/s)(398MiB/60001msec); 0 zone resets
    slat (usec): min=56, max=101177, avg=388.06, stdev=3492.74
    clat (nsec): min=1198, max=250992, avg=2091.27, stdev=3656.07
     lat (usec): min=58, max=101180, avg=390.57, stdev=3492.80
    clat percentiles (nsec):
     |  1.00th=[  1624],  5.00th=[  1672], 10.00th=[  1704], 20.00th=[  1736],
     | 30.00th=[  1768], 40.00th=[  1800], 50.00th=[  1848], 60.00th=[  1896],
     | 70.00th=[  1944], 80.00th=[  1976], 90.00th=[  2024], 95.00th=[  2064],
     | 99.00th=[  2704], 99.50th=[ 11968], 99.90th=[ 64768], 99.95th=[ 71168],
     | 99.99th=[102912]
   bw (  KiB/s): min= 3232, max= 9992, per=99.97%, avg=6797.47, stdev=1229.80, samples=238
   iops        : min=  808, max= 2498, avg=1699.33, stdev=307.47, samples=238
  lat (usec)   : 2=86.51%, 4=12.68%, 10=0.14%, 20=0.27%, 50=0.07%
  lat (usec)   : 100=0.30%, 250=0.02%, 500=0.01%, 750=0.01%
  cpu          : usr=2.60%, sys=8.55%, ctx=339403, majf=0, minf=2
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=237950,101989,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=15.5MiB/s (16.2MB/s), 15.5MiB/s-15.5MiB/s (16.2MB/s-16.2MB/s), io=929MiB (975MB), run=60001-60001msec
  WRITE: bw=6799KiB/s (6962kB/s), 6799KiB/s-6799KiB/s (6962kB/s-6962kB/s), io=398MiB (418MB), run=60001-60001msec

VZ virtiofs

Random Read
# fio /work/test.fio --section=random-read
random-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
...
fio-3.25
Starting 2 threads
Jobs: 2 (f=2): [r(2)][100.0%][r=30.4MiB/s][r=7775 IOPS][eta 00m:00s]
random-read: (groupid=0, jobs=2): err= 0: pid=3092: Fri Dec 30 12:38:17 2022
  read: IOPS=7747, BW=30.3MiB/s (31.7MB/s)(1816MiB/60001msec)
    slat (usec): min=101, max=3597, avg=254.73, stdev=38.78
    clat (nsec): min=1089, max=497831, avg=1781.28, stdev=1412.89
     lat (usec): min=103, max=3600, avg=256.94, stdev=38.99
    clat percentiles (nsec):
     |  1.00th=[ 1560],  5.00th=[ 1624], 10.00th=[ 1640], 20.00th=[ 1656],
     | 30.00th=[ 1672], 40.00th=[ 1688], 50.00th=[ 1704], 60.00th=[ 1736],
     | 70.00th=[ 1752], 80.00th=[ 1768], 90.00th=[ 1816], 95.00th=[ 1896],
     | 99.00th=[ 2128], 99.50th=[ 2704], 99.90th=[18816], 99.95th=[20864],
     | 99.99th=[50944]
   bw (  KiB/s): min=28656, max=31552, per=100.00%, avg=31024.42, stdev=178.01, samples=238
   iops        : min= 7164, max= 7888, avg=7756.10, stdev=44.50, samples=238
  lat (usec)   : 2=97.98%, 4=1.65%, 10=0.09%, 20=0.22%, 50=0.05%
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.01%
  cpu          : usr=1.62%, sys=6.58%, ctx=464919, majf=0, minf=4
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=464862,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=30.3MiB/s (31.7MB/s), 30.3MiB/s-30.3MiB/s (31.7MB/s-31.7MB/s), io=1816MiB (1904MB), run=60001-60001msec
Random Write
# fio /work/test.fio --section=random-write
random-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
...
fio-3.25
Starting 2 threads
Jobs: 2 (f=2): [w(2)][100.0%][w=15.9MiB/s][w=4081 IOPS][eta 00m:00s]
random-write: (groupid=0, jobs=2): err= 0: pid=3103: Fri Dec 30 12:39:18 2022
  write: IOPS=4542, BW=17.7MiB/s (18.6MB/s)(1065MiB/60001msec); 0 zone resets
    slat (usec): min=69, max=6989, avg=436.66, stdev=286.46
    clat (nsec): min=1060, max=3964.7k, avg=1924.20, stdev=7809.67
     lat (usec): min=70, max=6992, avg=439.02, stdev=286.79
    clat percentiles (nsec):
     |  1.00th=[ 1320],  5.00th=[ 1368], 10.00th=[ 1416], 20.00th=[ 1656],
     | 30.00th=[ 1720], 40.00th=[ 1752], 50.00th=[ 1768], 60.00th=[ 1800],
     | 70.00th=[ 1832], 80.00th=[ 1864], 90.00th=[ 1992], 95.00th=[ 2224],
     | 99.00th=[ 3792], 99.50th=[15808], 99.90th=[24448], 99.95th=[30848],
     | 99.99th=[61184]
   bw (  KiB/s): min= 6784, max=42992, per=100.00%, avg=18205.65, stdev=3022.13, samples=238
   iops        : min= 1696, max=10748, avg=4551.29, stdev=755.54, samples=238
  lat (usec)   : 2=90.37%, 4=8.69%, 10=0.31%, 20=0.31%, 50=0.30%
  lat (usec)   : 100=0.02%, 250=0.01%
  lat (msec)   : 4=0.01%
  cpu          : usr=1.07%, sys=9.36%, ctx=545082, majf=0, minf=2
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,272529,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=17.7MiB/s (18.6MB/s), 17.7MiB/s-17.7MiB/s (18.6MB/s-18.6MB/s), io=1065MiB (1116MB), run=60001-60001msec
Random Read/Write
# fio /work/test.fio --section=random-readwrite
random-readwrite: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
...
fio-3.25
Starting 2 threads
Jobs: 2 (f=2): [m(2)][100.0%][r=15.1MiB/s,w=6514KiB/s][r=3861,w=1628 IOPS][eta 00m:00s]
random-readwrite: (groupid=0, jobs=2): err= 0: pid=3114: Fri Dec 30 12:40:19 2022
  read: IOPS=3806, BW=14.9MiB/s (15.6MB/s)(892MiB/60001msec)
    slat (usec): min=36, max=14375, avg=342.60, stdev=179.88
    clat (nsec): min=1164, max=470611, avg=1933.66, stdev=1710.59
     lat (usec): min=38, max=14385, avg=345.00, stdev=180.00
    clat percentiles (nsec):
     |  1.00th=[ 1352],  5.00th=[ 1624], 10.00th=[ 1672], 20.00th=[ 1704],
     | 30.00th=[ 1736], 40.00th=[ 1752], 50.00th=[ 1784], 60.00th=[ 1816],
     | 70.00th=[ 1848], 80.00th=[ 1960], 90.00th=[ 2160], 95.00th=[ 2384],
     | 99.00th=[ 3120], 99.50th=[ 7200], 99.90th=[19840], 99.95th=[23680],
     | 99.99th=[51456]
   bw (  KiB/s): min= 8824, max=19064, per=100.00%, avg=15242.83, stdev=974.69, samples=238
   iops        : min= 2206, max= 4766, avg=3810.69, stdev=243.67, samples=238
  write: IOPS=1633, BW=6536KiB/s (6693kB/s)(383MiB/60001msec); 0 zone resets
    slat (usec): min=88, max=15049, avg=413.38, stdev=269.66
    clat (nsec): min=1041, max=93483, avg=1961.65, stdev=1520.59
     lat (usec): min=90, max=15058, avg=415.81, stdev=269.77
    clat percentiles (nsec):
     |  1.00th=[ 1320],  5.00th=[ 1400], 10.00th=[ 1576], 20.00th=[ 1736],
     | 30.00th=[ 1768], 40.00th=[ 1800], 50.00th=[ 1816], 60.00th=[ 1848],
     | 70.00th=[ 1896], 80.00th=[ 1976], 90.00th=[ 2160], 95.00th=[ 2416],
     | 99.00th=[ 3312], 99.50th=[14912], 99.90th=[20096], 99.95th=[25984],
     | 99.99th=[55552]
   bw (  KiB/s): min= 3736, max= 8208, per=100.00%, avg=6543.24, stdev=424.83, samples=238
   iops        : min=  934, max= 2052, avg=1635.80, stdev=106.21, samples=238
  lat (usec)   : 2=83.10%, 4=16.19%, 10=0.28%, 20=0.35%, 50=0.08%
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.01%
  cpu          : usr=1.34%, sys=6.75%, ctx=424596, majf=0, minf=2
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=228416,98040,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=14.9MiB/s (15.6MB/s), 14.9MiB/s-14.9MiB/s (15.6MB/s-15.6MB/s), io=892MiB (936MB), run=60001-60001msec
  WRITE: bw=6536KiB/s (6693kB/s), 6536KiB/s-6536KiB/s (6693kB/s-6693kB/s), io=383MiB (402MB), run=60001-60001msec

tarball 展開にかかる時間の比較

IOPS だけではちょっと実際の利用での差が分かりづらいのでそこそこのファイル含む tarball を展開するのにかかる時間を計測してみる。

linux-6.1.1.tar.xz を xz だけ展開した tar ファイルを展開する時間を計測してみる。 と思ったけど virtiofs 以外では時間がかかりすぎるので git の v2.39.0.tar.gz を使うことにした。gunzip 済みの tar ファイルを展開する時間を計測。(git の v2.39.0.tar はディレクトリを 213 個、ファイルを 4294 個含む 43MB のファイル)

ssh と 9p では owner の変更でたくさんエラー (Cannot change ownership to uid 0, gid 0: Permission denied) が出るので time tar -x --no-same-owner -f ../v2.39.0.tar と実行して計測。各環境で3回実行。

vz の virtiofs が圧倒的に速いですね。そして 9p は sshfs より速いとは言えない感じですね。vz が使えない環境では sshfs のままで良いのかな。

vmTypemountTypetime
QEMUreverse-sshfs0m15.758s
QEMUreverse-sshfs0m15.996s
QEMUreverse-sshfs0m15.586s
QEMU9p0m34.567s
QEMU9p0m29.186s
QEMU9p0m32.770s
VZvirtiofs0m6.130s
VZvirtiofs0m5.259s
VZvirtiofs0m5.370s

ちなみに macOS 上で直接同じコマンドで展開したところ約2秒でした。

Built with Hugo
テーマ StackJimmy によって設計されています。