Well, the previous NetBSD raid setup, was very slow.
Turns out, i had done some mistakes. But now, after ALOT of trial and error
speed is great, like 400mb/s write. Used to be like 30mb/s.

The below should be applicable to other BSD's running Softraid instead of Raidframe
aswell. TL;DR. sectPerSU=16 and a=32k was the resolution.

In raidctl(8) it says to use SectPerSU=16
But i thought that was invalid info, to larger drives, and or not my configuration
changes fully taking affect. I have tried ALOT of different settings, not seeing much
differences in the write speeds.
I started to dd if=/dev/zero of=/dev/rraid5 before reconfiguring the raid.

I had the below /etc/raid5.conf file as /etc/r5.conf during testing, to avoid it
being picked during a possible reboot.

/etc/raid5.conf

START array
# numRow numCol numSpare
1 3 1 

NAME=raid5@wd0
NAME=raid5@wd1
NAME=raid5@wd2

START spare
NAME=raid5@wd3

START layout
# sectPerSU SUsPerParityUnit SUsPerReconUnit RAID_level_5
16 1 1 5

START queue
fifo 127
	

These are the commands i ran:

# destroy possible previous raid5 config, also important while testing
gpt destroy raid5
raidctl -u raid5
dd if=/dev/zero of=/dev/rraid5 bs=1024k count=1
# This is maby not needed but did so anyway.

# **** means for each of the 4 disks, wd0,wd1,wd2,wd3
gpt destroy wd0 ****
gpt create -Af wd0 ****
gpt add -t raid -l raid5@wd0 -a 32k -b 64k wd0 ****
# start with small disks until you get the speed (add -s 1000g), then remove the size

raidctl -C /etc/raid5.conf raid5
raidctl -I 13371337 raid5
raidctl -iv raid5.
# this takes many hours, maby between 8 and 72+ hours, depending on raid size

gpt create -Af raid5
gpt add -a 32k -b 64k -t ffs -l island raid5
newfs -O2 -a 32k -b 64k -s -1g /dev/dk7  # reserve 1g for journal files
mount /dev/dk7 /mnt/island
     

Tried the above configs, but instead of 3 disk + 1 spare, i used 4 disks. Speeds were alot slower, so i stuck to 3 + 1 disks
4 disks

# dd if=/dev/zero bs=1024k count=1000 of=/mnt/island/test.txt
1048576000 bytes transferred in 17.498 secs (59925477 bytes/sec)
# dd if=/dev/zero bs=4096k count=1000 of=/mnt/island/test.txt
4194304000 bytes transferred in 122.339 secs (34284275 bytes/sec)
     

3 disks

# dd if=/dev/zero bs=1024k count=1000 of=/mnt/island/test.txt
1048576000 bytes transferred in 1.695 secs (618628908 bytes/sec)
# dd if=/dev/zero bs=4096k count=1000 of=/mnt/island/test.txt
4194304000 bytes transferred in 8.912 secs (470635547 bytes/sec)
# dd if=/dev/zero bs=4096k count=3000 of=/mnt/island/test.txt
12582912000 bytes transferred in 28.801 secs (436891496 bytes/sec)
     

I also tried to create a raid10, two raid1 (/dev/dk3, /dev/dk4) and (/dev/dk5, /dev/dk6)
Then connect them with a raid0, that uses the two raid1s, and reached similar speeds
to the final raid5...

All conversation captured in the netbsd-users mailing list
AI tried to help, but was ofcourse off. It was right about one thing though:
I was not crazy for attempting this!
Add this to your /etc/fstab

NAME=island     /mnt/island     ffs     log,rw           1 1
     

Dont use /dev/dkX in config files, since that may change during reboots.

Tips: Check dmesg after each command. When you create the raid with the raidctl -C,
it might complain with fatal errors, but they are ignored. that seems to be okey!

Raidctl -iv should take time. If instant, something is wrong!

To save time on the raidctl -iv, start with smaller disks, like 1TB.

If you get this error, if i remember correctly when you create the gpt for raid5,
you can clear that with: dd if=/dev/zero of=/dev/rraid5 bs=1024k count=1

gpt: /dev/rraid5: map entry doesn't fit media: new start + new size < start + size