Dave Pooser | 1 Mar 06:39 2011

Re: Format returning bogus controller info

On 2/28/11 4:23 PM, "Garrett D'Amore" <garrett <at> nexenta.com> wrote:

>Drives are ordered in the order they are *enumerated* when they *first*
>show up in the system.  *Ever*.

Is the same true of controllers? That is, will c12 remain c12 or
/pci <at> 0,0/pci8086,340c <at> 5 remain /pci <at> 0,0/pci8086,340c <at> 5 even if other
controllers are active?
--

-- 
Dave Pooser, ACSA
Manager of Information Services
Alford Media  http://www.alfordmedia.com
Moazam Raja | 1 Mar 07:38 2011
Picon

ZFS send/recv horribly slow on system with 1800+ filesystems

Hi all, I have a test system with a large amount of filesystems which
we take snapshots of and do send/recvs with.

On our test machine, we have 1800+ filesystems and about 5,000
snapshots.The system has 48GB of RAM, and 8 cores (x86). The
filesystem is comprised of 2 regular 1TB in a mirror with a 320GB
FusionIO flash card acting as a ZIL and read cache.

We've noticed that on systems with just a handful of filesystems, ZFS
send (recursive) is quite quick, but on our 1800+ fs box, it's
horribly slow.

For example,

root <at> testbox:~# zfs send -R chunky/0 <at> async-2011-02-28-15:11:20| pv -i
1 > /dev/null

2.51GB 0:04:57 [47.4kB/s] [            <=>                                     ]
^C

The other odd thing I've noticed is that during the 'zfs send' to
/dev/null, zpool iostat shows we're actually *writing* to the zpool at
the rate of 4MB-8MB/s, but reading almost nothing. How can this be the
case?

So I'm left with 2 questions -

1.) Does ZFS get immensely slow once we have thousands of filesystems?

2.) Why do we see 4MB-8MB/s of *writes* to the filesystem when we do a
(Continue reading)

Brandon High | 1 Mar 09:25 2011

Re: ZFS send/recv horribly slow on system with 1800+ filesystems

On Mon, Feb 28, 2011 at 10:38 PM, Moazam Raja <moazam <at> gmail.com> wrote:
> We've noticed that on systems with just a handful of filesystems, ZFS
> send (recursive) is quite quick, but on our 1800+ fs box, it's
> horribly slow.

When doing an incremental send, the system has to identify what blocks
have changed, which can take some time. If not much data has changed,
the delay can take longer than the actual send.

I've noticed that there's a small delay when starting a send of a new
snapshot and when starting the receive of one. Putting something like
mbuffer in the path helps to smooth things out. It won't help in the
example you've cited below, but it will help in real world use.

> The other odd thing I've noticed is that during the 'zfs send' to
> /dev/null, zpool iostat shows we're actually *writing* to the zpool at
> the rate of 4MB-8MB/s, but reading almost nothing. How can this be the
> case?

The writing seems odd, but the lack of reads doesn't. You might have
most or all of the data in the ARC or L2ARC, so your zpool doesn't
need to be read from.

> 1.) Does ZFS get immensely slow once we have thousands of filesystems?

No. Incremental sends might take longer, as I mentioned above.

> 2.) Why do we see 4MB-8MB/s of *writes* to the filesystem when we do a
> 'zfs send' to /dev/null ?

(Continue reading)

Brandon High | 1 Mar 09:32 2011

Re: Format returning bogus controller info

On Mon, Feb 28, 2011 at 9:39 PM, Dave Pooser <dave.zfs <at> alfordmedia.com> wrote:
> Is the same true of controllers? That is, will c12 remain c12 or
> /pci <at> 0,0/pci8086,340c <at> 5 remain /pci <at> 0,0/pci8086,340c <at> 5 even if other
> controllers are active?

You can rebuild the device tree if it bothers you. There are some
(outdated) instructions here:
http://spiralbound.net/blog/2005/12/21/rebuilding-the-solaris-device-tree
. I think you can do this all with a new boot environment, rather than
boot from a CD.

-B

--

-- 
Brandon High : bhigh <at> freaks.com
Roy Sigurd Karlsbakk | 1 Mar 17:03 2011
Picon

Good SLOG devices?

Hi

I'm running OpenSolaris 148 on a few boxes, and newer boxes are getting installed as we speak. What would you
suggest for a good SLOG device? It seems some new PCI-E-based ones are hitting the market, but will those
require special drivers? Cost is obviously alsoo an issue here....

Vennlige hilsener / Best regards

roy
--
Roy Sigurd Karlsbakk
(+47) 97542685
roy <at> karlsbakk.net
http://blogg.karlsbakk.net/
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ
for alle pedagoger å unngå eksessiv anvendelse av idiomer med fremmed opprinnelse. I de fleste
tilfeller eksisterer adekvate og relevante synonymer på norsk.

_______________________________________________
OpenIndiana-discuss mailing list
OpenIndiana-discuss <at> openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss
Ray Van Dolson | 1 Mar 17:08 2011

Re: Good SLOG devices?

On Tue, Mar 01, 2011 at 08:03:42AM -0800, Roy Sigurd Karlsbakk wrote:
> Hi
> 
> I'm running OpenSolaris 148 on a few boxes, and newer boxes are
> getting installed as we speak. What would you suggest for a good SLOG
> device? It seems some new PCI-E-based ones are hitting the market,
> but will those require special drivers? Cost is obviously alsoo an
> issue here....
> 
> Vennlige hilsener / Best regards
> 
> roy

What type of workload are you looking to handle?  We've had good luck
with pairs of Intel X-25E's for VM datastore duty.

We also have a DDRrive X1 which is probably the best option out there
currently and will handle workloads the X-25E's can't.

I believe a lot of folks here use the Vertex SLC-based SF-15 SSD's
also.

Ray
Cindy Swearingen | 1 Mar 17:10 2011
Picon

Re: Format returning bogus controller info

(Dave P...I sent this yesterday, but it bounced on your email address)

A small comment from me would be to create some test pools and replace
devices in the pools to see if device names remain the same or change
during these operations.

If the device names change and the pools are unhappy, retest similar
operations while the pools' are exported.

I've seen enough controller/device numbering wreak havoc on pool
availability that I'm automatically paranoid when I see the controller
numbering that you started with.

Thanks,

Cindy

On 02/28/11 22:39, Dave Pooser wrote:
> On 2/28/11 4:23 PM, "Garrett D'Amore" <garrett <at> nexenta.com> wrote:
> 
>> Drives are ordered in the order they are *enumerated* when they *first*
>> show up in the system.  *Ever*.
> 
> Is the same true of controllers? That is, will c12 remain c12 or
> /pci <at> 0,0/pci8086,340c <at> 5 remain /pci <at> 0,0/pci8086,340c <at> 5 even if other
> controllers are active?
Khushil Dep | 1 Mar 17:11 2011
Picon

Re: Good SLOG devices?

I'd back that. X25E's are great but also look at the STECH ZeusIOPS as well as the new Intel's.


---
W. A. Khushil Dep - khushil.dep <at> gmail.com -  07905374843
Windows - Linux - Solaris - ZFS - XenServer - FreeBSD - C/C++ - PHP/Perl - LAMP - Nexenta - Development - Consulting & Contracting




On 1 March 2011 16:08, Ray Van Dolson <rvandolson <at> esri.com> wrote:
On Tue, Mar 01, 2011 at 08:03:42AM -0800, Roy Sigurd Karlsbakk wrote:
> Hi
>
> I'm running OpenSolaris 148 on a few boxes, and newer boxes are
> getting installed as we speak. What would you suggest for a good SLOG
> device? It seems some new PCI-E-based ones are hitting the market,
> but will those require special drivers? Cost is obviously alsoo an
> issue here....
>
> Vennlige hilsener / Best regards
>
> roy

What type of workload are you looking to handle?  We've had good luck
with pairs of Intel X-25E's for VM datastore duty.

We also have a DDRrive X1 which is probably the best option out there
currently and will handle workloads the X-25E's can't.

I believe a lot of folks here use the Vertex SLC-based SF-15 SSD's
also.

Ray
_______________________________________________
zfs-discuss mailing list
zfs-discuss <at> opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

_______________________________________________
zfs-discuss mailing list
zfs-discuss <at> opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Roy Sigurd Karlsbakk | 1 Mar 18:56 2011
Picon

Re: [zfs-discuss] Good SLOG devices?

> a) do you need an SLOG at all? Some workloads (asynchronous ones) will
> never benefit from an SLOG.

We're planning to use this box for CIFS/NFS, so we'll need an SLOG to speed things up.

> b) form factor. at least one manufacturer uses a PCIe card which is
> not compliant with the PCIe form-factor and will not fit in many cases
> -- especially typical 1U boxes.

The box is 4U with some 7 8x PCIe slots, so I think it should do fine

> c) driver support.

That was why I asked here in the first place...

> d) do they really just go straight to ram/flash, or do they have an
> on-device SAS or SATA bus? Some PCIe devices just stick a small flash
> device on a SAS or SATA controller. I suspect that those devices won't
> see a lot of benefit relative to an external drive (although they
> could theoretically drive that private SAS/SATA bus at much higher rates
> than an external bus -- but I've not checked into it.)
> 
> The other thing with PCIe based devices is that they consume an IO
> slot,
> which may be precious to you depending on your system board and other
> I/O needs.

As I mentioned above, we have sufficient slots. As for the SATA/SAS onboard controller, that was the reason
I asked here in the first place.

So - do anyone know a good device for this? X25-E is rather old now, so there should be better ones available......

Vennlige hilsener / Best regards

roy
--
Roy Sigurd Karlsbakk
(+47) 97542685
roy <at> karlsbakk.net
http://blogg.karlsbakk.net/
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ
for alle pedagoger å unngå eksessiv anvendelse av idiomer med fremmed opprinnelse. I de fleste
tilfeller eksisterer adekvate og relevante synonymer på norsk.

_______________________________________________
OpenIndiana-discuss mailing list
OpenIndiana-discuss <at> openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss
Ray Van Dolson | 1 Mar 19:04 2011

Re: Good SLOG devices?

On Tue, Mar 01, 2011 at 09:56:35AM -0800, Roy Sigurd Karlsbakk wrote:
> > a) do you need an SLOG at all? Some workloads (asynchronous ones) will
> > never benefit from an SLOG.
> 
> We're planning to use this box for CIFS/NFS, so we'll need an SLOG to
> speed things up.
>  
> > b) form factor. at least one manufacturer uses a PCIe card which is
> > not compliant with the PCIe form-factor and will not fit in many cases
> > -- especially typical 1U boxes.
> 
> The box is 4U with some 7 8x PCIe slots, so I think it should do fine
> 
> > c) driver support.
> 
> That was why I asked here in the first place...
> 
> > d) do they really just go straight to ram/flash, or do they have an
> > on-device SAS or SATA bus? Some PCIe devices just stick a small flash
> > device on a SAS or SATA controller. I suspect that those devices won't
> > see a lot of benefit relative to an external drive (although they
> > could theoretically drive that private SAS/SATA bus at much higher rates
> > than an external bus -- but I've not checked into it.)
> > 
> > The other thing with PCIe based devices is that they consume an IO
> > slot,
> > which may be precious to you depending on your system board and other
> > I/O needs.
> 
> As I mentioned above, we have sufficient slots. As for the SATA/SAS
> onboard controller, that was the reason I asked here in the first
> place.
> 
> So - do anyone know a good device for this? X25-E is rather old now,
> so there should be better ones available......

I think the OCZ Vertex 2 EX (SLC) is fairly highly regarded:

    http://www.ocztechnology.com/ocz-vertex-2-ex-series-sata-ii-2-5-ssd.html

Note that if you're using an LSI backplane (probably are if you're
using SuperMicro hardware), they have tended to certify only against
the X-25E.  Other drives should work fine, but just an FYI.

This page (maybe a little dated, I'm not sure) has some pretty good
info:

    http://www.nexenta.org/projects/site/wiki/About_suggested_NAS_SAN_Hardware

> 
> Vennlige hilsener / Best regards
> 
> roy

Ray

Gmane