Emmanuel Dreyfus | 30 Sep 15:44 2014
X-Face
Picon

[PATCH] GOP_ALLOC and fallocate for PUFFS

Hello

When a PUFFS filesystem uses the page cache, data enters the
cache with no guarantee it will be flushed. If it cannot be flushed
(bcause PUFFS write requests get EDQUOT or ENOSPC), then the
kernel will loop forever trying to flush data from the cache,
and the filesystem cannot be unmounted without -f (and data loss).

In the attached patch, I add in PUFFS:
- support for the fallocate operation 
- a puffs_gop_alloe() function that use fallocate
- when writing through the page cache we call first GOP_ALLOC to make
  sure backend storage is allocated for the data we cache. debug printf
  show a sane behavior, GOP_ALLOC calling puffs_gop_alloc only when required.

If the filesystem does not implement fallocate, we keep the current 
behavior of filling the page cache with data we are not sure we can flush. 
Perhaps we can improve further: missing fallocate can be emulated by 
writing zeroed chuncks. I have implemented that in libperfuse, but 
we may want to have this in libpuffs, enabled by a mount option. Input
welcome.

--

-- 
Emmanuel Dreyfus
manu <at> netbsd.org
Index: lib/libpuffs/dispatcher.c
===================================================================
RCS file: /cvsroot/src/lib/libpuffs/dispatcher.c,v
(Continue reading)

Emmanuel Dreyfus | 25 Sep 10:43 2014
X-Face
Picon

syscall stub

Hi

I need to add a layer in libc for posix_allocate() because the
system call argument list does not match the prototype exposed
to userland. 

How can I do it? My problem is that the system call and the libc
function have the same name. Do I need to rename the system call,
to something like _posix_fallocate ? Or is there a trick to do
it otherwise.

--

-- 
Emmanuel Dreyfus
manu <at> netbsd.org

Emmanuel Dreyfus | 25 Sep 03:19 2014
Picon

[PATCH] fallocate() for FFS

Implementing fallocate for FFS seems quite obvious.  Is there anything I
missed in the patch below? Is it good enough to commit?

This lets me discover that posix_fallocate() libc stub is buggy. Using
it through #include <unistd.h> leads argument corruption. Defining it as
int      posix_fallocate(int, int, off_t, off_t); let it work (note the
second int pad, as in syscalls.master). I guess we need to add a libc
stub too.

Index: sys/ufs/ffs/ffs_vnops.c
===================================================================
RCS file: /cvsroot/src/sys/ufs/ffs/ffs_vnops.c,v
retrieving revision 1.125
diff -U 4 -r1.125 ffs_vnops.c
--- sys/ufs/ffs/ffs_vnops.c     25 Jul 2014 08:20:53 -0000      1.125
+++ sys/ufs/ffs/ffs_vnops.c     25 Sep 2014 01:09:50 -0000
 <at>  <at>  -114,9 +114,9  <at>  <at> 
        { &vop_getattr_desc, ufs_getattr },             /* getattr */
        { &vop_setattr_desc, ufs_setattr },             /* setattr */
        { &vop_read_desc, ffs_read },                   /* read */
        { &vop_write_desc, ffs_write },                 /* write */
-       { &vop_fallocate_desc, genfs_eopnotsupp },      /* fallocate */
+       { &vop_fallocate_desc, ffs_fallocate },         /* fallocate */
        { &vop_fdiscard_desc, genfs_eopnotsupp },       /* fdiscard */
        { &vop_ioctl_desc, ufs_ioctl },                 /* ioctl */
        { &vop_fcntl_desc, ufs_fcntl },                 /* fcntl */
        { &vop_poll_desc, ufs_poll },                   /* poll */
 <at>  <at>  -326,8 +326,41  <at>  <at> 
        return error;
 }
(Continue reading)

Emmanuel Dreyfus | 24 Sep 13:33 2014
Picon

GOP_ALLOC vs VOP_FALLOCATE

Hi

Is there a real difference between GOP_ALLOC and VOP_FALLOCATE? The two
operation seem very similar, with just extra flag and cred args for
GOP_ALLOC. 

--

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu <at> netbsd.org

Robert Swindells | 22 Sep 20:15 2014
Picon

Re HDMI transmitter interface


Manuel Bouyer wrote:
>I made some progress on the TDA19988 HDMI transmitter driver as
>found in the beaglebone black (doens't work yet, sorry :).
>The driver will attach directly to the i2c bus, because other
>devices are also connected to this bus (eeprom, the power managenent IC, and
>more depending on connected capes). Then the TDA19988 driver and the
>video drivers have to talk together: the TDA19988 driver knows when a
>monitor is connected/disconnected and also can read the EDID, and
>the video driver knows which mode to select.

Had you considered using the drmkms code for this SoC ?

Current Linux sources have a driver for the display and for this HDMI
transmitter.

They were added to Linux after the last import that riastradh <at>  made so
are not in the NetBSD tree yet.

Robert Swindells

Manuel Bouyer | 22 Sep 17:55 2014

HDMI transmitter interface

Hello,
I made some progress on the TDA19988 HDMI transmitter driver as
found in the beaglebone black (doens't work yet, sorry :).
The driver will attach directly to the i2c bus, because other
devices are also connected to this bus (eeprom, the power managenent IC, and
more depending on connected capes). Then the TDA19988 driver and the
video drivers have to talk together: the TDA19988 driver knows when a
monitor is connected/disconnected and also can read the EDID, and
the video driver knows which mode to select.

The kernel config looks like this:
tifb*   at obio0 addr 0x4830E000 size 0x1000 intr 36
# tda19988 HDMI transmitter (beagelbone black)
tda19988hdmi0   at iic0 addr 0x70 #also uses addr 0x34
options         TIFB_TRANSMITTER="\"tda19988hdmi0\""
options         TIFB_TRANSMITTER_BPP=16

TIFB_TRANSMITTER and TIFB_TRANSMITTER_BPP are used by the tifb driver
only. the string in TIFB_TRANSMITTER allows tifb(4) to do a
device_find_by_xname() to find the tda19988hdmi0's device_t
(at this time this means that iic0 has to be attached first).

tifb(4) and tda19988hdmi(4) shares a structure declared in
sys/dev/videomode/video_transmitter.h (new file):
struct video_transmitter {
        device_t        vdt_t_dev; /* video transmitter device */
        device_t        vdt_c_dev; /* video controller device */
        /*
         * data we got from the monitor. Can be filled either by the
         * transmitter or the controller driver
(Continue reading)

Emmanuel Dreyfus | 22 Sep 06:28 2014
Picon

How PUFFS should deal with EDQUOT?

Hi

When a PUFFS filesystem enforces quota, a process doing a write over
quota will end frozen in DE+ state.

The problem is that  we have written data in the page cache that is
supposed to go to disk. The code path is a bit complicated, but
basically we go in genfs VOP_PUTPAGE, which leads to genfs_do_io() where
we have a VOP_STRATEGY, which cause PUFFS write. The PUFFS write will
get EDQUOT, but genfs_do_io()  ignores VOP_STRATEGY's return value and
retries forever.

In other words, when flushing the cache, the kernel ignores errors from
the filesystem and runs an endless loop attempting to flush data, during
which the process that did the over quota write is not allowed to
complete exit().

What is the proper way to deal with that? Is it reasonable to wipe the
page cache using puffs_inval_pagecache_node() when write gets a failure?
Any failure? Or just EDQUOT and ENOSPC? Should that happen in libpuffs
or in the filesystem (libperfuse here)?

--

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu <at> netbsd.org

bycn82 | 21 Sep 02:08 2014
Picon

build kernel from source

Hi All,

Help, I am just download the source from CVS using this command `cvs checkout -A -P src`,
so it should be current development version, But I met two issues.

I am new here:) please help.

Try to build kernel from source

Command:
cd /usr/src; ./build.sh -O /usr/obj -U -j 8 tools kernel=NB6 modules distribution sets


Result:
configure: creating ./config.status
config.status: creating host-mkdep
chmod +x host-mkdep
#   install  /tooldir.NetBSD-6.1.4-amd64/bin/nbhost-mkdep
mkdir -p /tooldir.NetBSD-6.1.4-amd64/bin
/usr/src/tools/binstall/xinstall -c  -r -m 555 host-mkdep /tooldir.NetBSD-6.1.4-amd64/bin/nbhost-mkdep
make: exec(/usr/src/tools/binstall/xinstall) failed (No such file or directory)
*** Error code 1

Stop.
make: stopped in /usr/src/tools/host-mkdep
*** Error code 1

Actually the xinstall is not in the folder
# find / -name xinstall
/usr/src/usr.bin/xinstall
/usr/obj/tools/binstall/xinstall
# cat ~/build_kernel

Try to build the npfctl command

# cd /usr/src/usr.sbin/npf/npfctl/
# pwd
/usr/src/usr.sbin/npf/npfctl
# make
#       lex  npfctl/npf_scan.c
/usr/src/tooldir.NetBSD-6.1.4-amd64/bin/nblex    -onpf_scan.c npf_scan.l
make: exec(/usr/src/tooldir.NetBSD-6.1.4-amd64/bin/nblex) failed (No such file or directory)
*** Error code 1

Stop.
make: stopped in /usr/src/usr.sbin/npf/npfctl
#

How to change the NetBSD6.1.4? I am using the current development version of source!
Maxime Villard | 20 Sep 20:48 2014
Picon

Brainy: Set of 33 potential bugs

Hi,
here is another set of 33 potential bugs found by my code scanner.

	http://m00nbsd.net/ae123a9bae03f7dde5c6d654412daf5a.html#Report-3

Not all bugs are listed here; I've put only those which looked like proper
bugs. I guess they will all need to be fixed in NetBSD-7.

Maxime

Takahiro HAYASHI | 19 Sep 13:06 2014
Picon

patch: verbose debug code for xhci

Hello,

This patch does NOT improve xhci.c at all but may (or not) help
people who try to read debugging hexdump even though they are not
familiar with it.
This patch is imcomplete, buggy, and does not support all requests.

For example, you can read TRB values in human-readable form like this:

Before:
xhci0: xhci_do_command: input: 0x000000000355e000 0x00000000 0x01002c00
xhci0: xhci_do_command: output: 0x00000000033aa010 0x01000000 0x01008401

After:
xhci0: xhci_do_command: input: 0x000000000355e000 0x00000000 0x01002c00
TYPE_ADDRESS_DEVICE(11) slot 1 bsr 0 ictx 000000000355e000 c 0
xhci0: xhci_do_command: output: 0x00000000033aa010 0x01000000 0x01008401
EVENT_CMD_COMPLETE(33) SUCCESS(1) slot 1 vf 0 c 1 param 0 cmd 00000000033aa010

by inserting xhci_dump_trb() to appropriate positions.

--- xhci.c.orig	2014-08-12 23:29:37.000000000 +0900
+++ xhci.c	2014-09-19 08:04:11.000000000 +0900
 <at>  <at>  -1734,6 +2451,7  <at>  <at>  xhci_do_command(struct xhci_softc * cons
  	device_printf(sc->sc_dev, "%s input: "
  	    "0x%016"PRIx64" 0x%08"PRIx32" 0x%08"PRIx32"\n", __func__,
  	    trb->trb_0, trb->trb_2, trb->trb_3);
+	xhci_dump_trb(-1, trb);

  	mutex_enter(&sc->sc_lock);

 <at>  <at>  -1759,6 +2477,7  <at>  <at>  xhci_do_command(struct xhci_softc * cons
  	device_printf(sc->sc_dev, "%s output: "
  	    "0x%016"PRIx64" 0x%08"PRIx32" 0x%08"PRIx32"\n", __func__,
  	    trb->trb_0, trb->trb_2, trb->trb_3);
+	xhci_dump_trb(-1, trb);

  	switch (XHCI_TRB_2_ERROR_GET(trb->trb_2)) {
  	case XHCI_TRB_ERROR_SUCCESS:

For example, you can see device request and reply in human-readable form
like this:

Before:
req: 80 06 0200 0000 0009
req: 80 06 0200 0000 001f

After:
req: GET_DESCRIPTOR(6) IN,STAND,DEV CONFIG(0x0200) idx 0000 len 9
getdesc: len 9: CONFIG(2) len 9<len 31 noiface 1 confval 1 iconf 0 attr e0 pwr 0>
req: GET_DESCRIPTOR(6) IN,STAND,DEV CONFIG(0x0200) idx 0000 len 31
getdesc: len 31: CONFIG(2) len 9<len 31 noiface 1 confval 1 iconf 0 attr e0 pwr 0>, INTERFACE(4) len 9<>,
ENDPOINT(5) len 7<notyet>, SSEP_COMPANION(48) len 6<maxburst 0 attr 00 b/i 2>

by adding xhci_dump_{req,reply}().

--- xhci.c.orig	2014-08-12 23:29:37.000000000 +0900
+++ xhci.c	2014-09-19 08:04:11.000000000 +0900
 <at>  <at>  -1235,6 +1801,10  <at>  <at>  xhci_handle_event(struct xhci_softc * co
  		}
  		xfer->status = err;

+		if (err == USBD_NORMAL_COMPLETION && (trb_0 & 0x3) == 0x3) {
+			xhci_dump_reply(0, xfer);
+		}
+
  		//mutex_enter(&sc->sc_lock); /* XXX ??? */
  		if ((trb_3 & XHCI_TRB_3_ED_BIT) != 0) {
  			if ((trb_0 & 0x3) == 0x0) {
 <at>  <at>  -2541,10 +3491,8  <at>  <at>  xhci_device_ctrl_start(usbd_xfer_handle
  	uint32_t control;
  	u_int i;

-	DPRINTF(("%s\n", __func__));
-	DPRINTF(("req: %02x %02x %04x %04x %04x\n", req->bmRequestType,
-	    req->bRequest, UGETW(req->wValue), UGETW(req->wIndex),
-	    UGETW(req->wLength)));
+	DPRINTF(("req: ));
+	xhci_dump_req(0, req);

  	/* XXX */
  	if (tr->is_halted) {

Thanks,
-- 
t-hash

--- xhci.c.orig	2014-08-12 23:29:37.000000000 +0900
+++ xhci.c	2014-09-19 08:04:11.000000000 +0900
 <at>  <at>  -147,6 +234,29  <at>  <at>  static void xhci_device_bulk_done(usbd_x
 static void xhci_timeout(void *);
 static void xhci_timeout_task(void *);

+#ifdef XHCI_DEBUG
+static const char *str_trberr(int);
+static void xhci_dump_ictlctx(int, uint32_t *);
+static void xhci_dump_sctx(int, uint32_t *);
+static void xhci_dump_epctx(int, uint32_t *);
+static void xhci_dump_trbn(int, uint64_t, uint32_t, uint32_t);
+static void xhci_dump_trb(int, const struct xhci_trb * const);
+static void xhci_dump_portsc(int, uint32_t);
+static void xhci_dump_desc(int, void *);
+static void xhci_dump_reply(int, usbd_xfer_handle);
+static void xhci_dump_req(int, void *);
+#else /* XHCI_DEBUG */
+#define xhci_dump_ictlctx(n, x) ((void)0)
+#define xhci_dump_sctx(n, x) ((void)0)
+#define xhci_dump_epctx(n,x) ((void)0)
+#define xhci_dump_trbn(n, t0, t2, t3) ((void)0)
+#define xhci_dump_trb(n, x) ((void)0)
+#define xhci_dump_portsc(n, x) ((void)0)
+#define xhci_dump_desc(n, x) ((void)0)
+#define xhci_dump_reply(n, x) ((void)0)
+#define xhci_dump_req(n, x) ((void)0)
+#endif /* XHCI_DEBUG */
+
 static const struct usbd_bus_methods xhci_bus_methods = {
 	.open_pipe = xhci_open,
 	.soft_intr = xhci_softintr,
 <at>  <at>  -2934,3 +4100,753  <at>  <at>  xhci_timeout_task(void *addr)
 #endif
 	mutex_exit(&sc->sc_lock);
 }
+
+
+#ifdef XHCI_DEBUG
+/* verbose debug dump */
+
+struct tbl_t {
+	int		idx;
+	const char	*str;
+};
+
+#define TBLELEM(elem) { .idx = (elem), .str = #elem }
+
+static struct tbl_t trbtypetbl[] = {
+	TBLELEM(XHCI_TRB_TYPE_RESERVED),
+	TBLELEM(XHCI_TRB_TYPE_NORMAL),
+	TBLELEM(XHCI_TRB_TYPE_SETUP_STAGE),
+	TBLELEM(XHCI_TRB_TYPE_DATA_STAGE),
+	TBLELEM(XHCI_TRB_TYPE_STATUS_STAGE),
+	TBLELEM(XHCI_TRB_TYPE_ISOCH),
+	TBLELEM(XHCI_TRB_TYPE_LINK),
+	TBLELEM(XHCI_TRB_TYPE_EVENT_DATA),
+	TBLELEM(XHCI_TRB_TYPE_NOOP),
+	TBLELEM(XHCI_TRB_TYPE_ENABLE_SLOT),
+	TBLELEM(XHCI_TRB_TYPE_DISABLE_SLOT),
+	TBLELEM(XHCI_TRB_TYPE_ADDRESS_DEVICE),
+	TBLELEM(XHCI_TRB_TYPE_CONFIGURE_EP),
+	TBLELEM(XHCI_TRB_TYPE_EVALUATE_CTX),
+	TBLELEM(XHCI_TRB_TYPE_RESET_EP),
+	TBLELEM(XHCI_TRB_TYPE_STOP_EP),
+	TBLELEM(XHCI_TRB_TYPE_SET_TR_DEQUEUE),
+	TBLELEM(XHCI_TRB_TYPE_RESET_DEVICE),
+	TBLELEM(XHCI_TRB_TYPE_FORCE_EVENT),
+	TBLELEM(XHCI_TRB_TYPE_NEGOTIATE_BW),
+	TBLELEM(XHCI_TRB_TYPE_SET_LATENCY_TOL),
+	TBLELEM(XHCI_TRB_TYPE_GET_PORT_BW),
+	TBLELEM(XHCI_TRB_TYPE_FORCE_HEADER),
+	TBLELEM(XHCI_TRB_TYPE_NOOP_CMD),
+	TBLELEM(XHCI_TRB_EVENT_TRANSFER),
+	TBLELEM(XHCI_TRB_EVENT_CMD_COMPLETE),
+	TBLELEM(XHCI_TRB_EVENT_PORT_STS_CHANGE),
+	TBLELEM(XHCI_TRB_EVENT_BW_REQUEST),
+	TBLELEM(XHCI_TRB_EVENT_DOORBELL),
+	TBLELEM(XHCI_TRB_EVENT_HOST_CTRL),
+	TBLELEM(XHCI_TRB_EVENT_DEVICE_NOTIFY),
+	TBLELEM(XHCI_TRB_EVENT_MFINDEX_WRAP),
+};
+
+static struct tbl_t trberrtbl[] = {
+	TBLELEM(XHCI_TRB_ERROR_INVALID),
+	TBLELEM(XHCI_TRB_ERROR_SUCCESS),
+	TBLELEM(XHCI_TRB_ERROR_DATA_BUF),
+	TBLELEM(XHCI_TRB_ERROR_BABBLE),
+	TBLELEM(XHCI_TRB_ERROR_XACT),
+	TBLELEM(XHCI_TRB_ERROR_TRB),
+	TBLELEM(XHCI_TRB_ERROR_STALL),
+	TBLELEM(XHCI_TRB_ERROR_RESOURCE),
+	TBLELEM(XHCI_TRB_ERROR_BANDWIDTH),
+	TBLELEM(XHCI_TRB_ERROR_NO_SLOTS),
+	TBLELEM(XHCI_TRB_ERROR_STREAM_TYPE),
+	TBLELEM(XHCI_TRB_ERROR_SLOT_NOT_ON),
+	TBLELEM(XHCI_TRB_ERROR_ENDP_NOT_ON),
+	TBLELEM(XHCI_TRB_ERROR_SHORT_PKT),
+	TBLELEM(XHCI_TRB_ERROR_RING_UNDERRUN),
+	TBLELEM(XHCI_TRB_ERROR_RING_OVERRUN),
+	TBLELEM(XHCI_TRB_ERROR_VF_RING_FULL),
+	TBLELEM(XHCI_TRB_ERROR_PARAMETER),
+	TBLELEM(XHCI_TRB_ERROR_BW_OVERRUN),
+	TBLELEM(XHCI_TRB_ERROR_CONTEXT_STATE),
+	TBLELEM(XHCI_TRB_ERROR_NO_PING_RESP),
+	TBLELEM(XHCI_TRB_ERROR_EV_RING_FULL),
+	TBLELEM(XHCI_TRB_ERROR_INCOMPAT_DEV),
+	TBLELEM(XHCI_TRB_ERROR_MISSED_SERVICE),
+	TBLELEM(XHCI_TRB_ERROR_CMD_RING_STOP),
+	TBLELEM(XHCI_TRB_ERROR_CMD_ABORTED),
+	TBLELEM(XHCI_TRB_ERROR_STOPPED),
+	TBLELEM(XHCI_TRB_ERROR_LENGTH),
+	TBLELEM(XHCI_TRB_ERROR_BAD_MELAT),
+	TBLELEM(XHCI_TRB_ERROR_ISOC_OVERRUN),
+	TBLELEM(XHCI_TRB_ERROR_EVENT_LOST),
+	TBLELEM(XHCI_TRB_ERROR_UNDEFINED),
+	TBLELEM(XHCI_TRB_ERROR_INVALID_SID),
+	TBLELEM(XHCI_TRB_ERROR_SEC_BW),
+	TBLELEM(XHCI_TRB_ERROR_SPLIT_XACT),
+};
+
+#define TBLLOOKUP(tbl,skip) do {					\
+		int i;							\
+		for (i = 0; i < __arraycount(tbl) ; i++)		\
+			if (tbl[i].idx == key)				\
+				return &tbl[i].str[skip];		\
+		return "unknown";					\
+	} while (0/*CONSTCOND*/)
+
+static const char *
+str_trbtype(int key)
+{
+	TBLLOOKUP(trbtypetbl, 9);
+}
+
+static const char *
+str_trberr(int key)
+{
+	TBLLOOKUP(trberrtbl, 15);
+}
+
+static void __used
+xhci_dump_ictlctx(int dbglvl, uint32_t *cp)
+{
+	int i, n;
+
+	if (xhcidebug <= dbglvl)
+		return;
+
+	printf("ctlctx Drop: ");
+	for (i = 0, n = 0; i < 32; i++)
+		if ((cp[0] >> i) & 1) {
+			printf("%d ", i);
+			n++;
+		}
+	if (n == 0)
+		printf("(none)");
+	printf("\nctlctx Add: ");
+	for (i = 0, n = 0; i < 32; i++)
+		if ((cp[1] >> i) & 1) {
+			printf("%d ", i);
+			n++;
+		}
+	if (n == 0)
+		printf("(none)");
+	printf("\n");
+}
+
+static void __used
+xhci_dump_sctx(int dbglvl, uint32_t *cp)
+{
+	if (xhcidebug <= dbglvl)
+		return;
+
+	printf("cp0: ctxentry %u hub %u mtt %u speed %u rtstr %05x\n",
+	    XHCI_SCTX_0_CTX_NUM_GET(cp[0]),
+	    XHCI_SCTX_0_HUB_GET(cp[0]),
+	    XHCI_SCTX_0_MTT_GET(cp[0]),
+	    XHCI_SCTX_0_SPEED_GET(cp[0]),
+	    XHCI_SCTX_0_ROUTE_GET(cp[0]));
+	printf("cp1: nports %u rhport %u maxel %u\n",
+	    XHCI_SCTX_1_NUM_PORTS_GET(cp[1]),
+	    XHCI_SCTX_1_RH_PORT_GET(cp[1]),
+	    XHCI_SCTX_1_MAX_EL_GET(cp[1]));
+	printf("cp2: intrtarg %u ttt %u ttportnum %u tthubsid %u\n",
+	    XHCI_SCTX_2_IRQ_TARGET_GET(cp[2]),
+	    XHCI_SCTX_2_TT_THINK_TIME_GET(cp[2]),
+	    XHCI_SCTX_2_TT_PORT_NUM_GET(cp[2]),
+	    XHCI_SCTX_2_TT_HUB_SID_GET(cp[2]));
+	printf("cp3: slotstate %u addr %u\n",
+	    XHCI_SCTX_3_SLOT_STATE_GET(cp[3]),
+	    XHCI_SCTX_3_DEV_ADDR_GET(cp[3]));
+}
+
+#ifdef notyet
+static void
+xhci_dump_stctx(int dbglvl, uint32_t *cp)
+{
+}
+#endif
+
+static void __used
+xhci_dump_epctx(int dbglvl, uint32_t *cp)
+{
+	if (xhcidebug <= dbglvl)
+		return;
+
+	char sbuf[126];
+	snprintb(sbuf, sizeof sbuf, "\177\020"
+		"f\030\010MESIT\0f\020\010IVAL\0b\017LSA\0"
+		"f\012\010MAXPS\0f\010\2MULT\0f\0\3EPSTATE\0\0", cp[0]);
+	printf("cp0: %s\n", sbuf);
+
+	snprintb(sbuf, sizeof sbuf, "\177\020"
+		"f\020\020MPS\0f\010\010MBURST\0b\7HID\0"
+		"f\3\3EPTYPE\0f\1\2CERR\0\0", cp[1]);
+	printf("cp1: %s\n", sbuf);
+
+	printf("cp2: TR dqptr %016"PRIx64"\n",
+		(uint64_t)cp[3] << 32 | cp[2]);
+
+	printf("cp4: mesit %016x avgTRBlen %u\n",
+		XHCI_EPCTX_4_MAX_ESIT_PAYLOAD_GET(cp[4]),
+		XHCI_EPCTX_4_AVG_TRB_LEN_GET(cp[4]));
+}
+
+#ifdef notyet
+static void
+xhci_dump_devctx(int dbglvl, uint32_t *cp)
+{
+}
+
+static void
+xhci_dump_pbwctx(int dbglvl, uint32_t *cp)
+{
+}
+#endif
+
+#define _B(x) ((x) ? 1 : 0)
+
+static void __used
+xhci_dump_trbn(int dbglvl, uint64_t trb_0, uint32_t trb_2, uint32_t trb_3)
+{
+	uint32_t trbtype;
+	char sbuf[128];
+
+	if (xhcidebug <= dbglvl)
+		return;
+
+	trb_0 = le64toh(trb_0);
+	trb_2 = le32toh(trb_2);
+	trb_3 = le32toh(trb_3);
+
+	trbtype = XHCI_TRB_3_TYPE_GET(trb_3);
+	printf("%s(%u) ", str_trbtype(trbtype), trbtype);
+
+	switch(trbtype) {
+	case XHCI_TRB_TYPE_SETUP_STAGE:
+		snprintb(sbuf, sizeof sbuf, "\177\020"
+			"f\020\020wValue\0f\010\010bRequest\0"
+			"f\0\010bmReqType\0", (trb_0 & 0xffffffffU));
+		printf("%s ", sbuf);
+		snprintb(sbuf, sizeof sbuf, "\177\020"
+			"f\020\020wLength\0f\0\020wIndex\0\0", (trb_0 >> 32));
+		printf("%s ", sbuf);
+		snprintb(sbuf, sizeof sbuf, "\177\020"
+			"b\020TRT\0b\6IDT\0b\5IOC\0b\0C\0\0",
+			trb_3);
+		printf("irq %u tdsz %u len %u %s\n",
+			XHCI_TRB_2_IRQ_GET(trb_2),
+			XHCI_TRB_2_TDSZ_GET(trb_2),
+			XHCI_TRB_2_BYTES_GET(trb_2),
+			sbuf);
+		break;
+	case XHCI_TRB_TYPE_ISOCH:
+		snprintb(sbuf, sizeof sbuf, "\177\020"
+			"b\037SIA\0f\024\013FRAMEID\0f\020\4TLBPC\0"
+			"b\011BEI\0f\7\2TBC\0"
+			"b\6IDT\0"
+			"b\5IOC\0b\4CH\0"
+			"b\3NS\0b\2ISP\0"
+			"b\1ENT\0b\0C\0\0",
+			trb_3);
+		printf("irq %u tdsz %u len %u %s buf %016"PRIx64"\n",
+			XHCI_TRB_2_IRQ_GET(trb_2),
+			XHCI_TRB_2_TDSZ_GET(trb_2),
+			XHCI_TRB_2_BYTES_GET(trb_2),
+			sbuf,
+			trb_0);
+		break;
+	case XHCI_TRB_TYPE_LINK:
+		snprintb(sbuf, sizeof sbuf, "\177\020"
+			"b\5IOC\0b\4CH\0b\1TC\0b\0C\0\0",
+			trb_3);
+		printf("irq %u %s ringseg %016"PRIx64"\n",
+			XHCI_TRB_2_IRQ_GET(trb_2),
+			sbuf,
+			trb_0);
+		break;
+	case XHCI_TRB_TYPE_NORMAL:
+	case XHCI_TRB_TYPE_DATA_STAGE:
+	case XHCI_TRB_TYPE_STATUS_STAGE:
+	case XHCI_TRB_TYPE_EVENT_DATA:
+		snprintb(sbuf, sizeof sbuf, "\177\020"
+			"b\020DIR\0"
+			"b\6IDT\0"
+			"b\5IOC\0b\4CH\0"
+			"b\3NS\0b\2ISP\0"
+			"b\1ENT\0b\0C\0\0",
+			trb_3);
+		printf("irq %u tdsz %u len %u %s buf %016"PRIx64"\n",
+			XHCI_TRB_2_IRQ_GET(trb_2),
+			XHCI_TRB_2_TDSZ_GET(trb_2),
+			XHCI_TRB_2_BYTES_GET(trb_2),
+			sbuf,
+			trb_0);
+		break;
+	case XHCI_TRB_TYPE_RESERVED:
+	case XHCI_TRB_TYPE_NOOP:
+	case XHCI_TRB_TYPE_ENABLE_SLOT:
+		printf("\n");
+		break;
+	case XHCI_TRB_TYPE_DISABLE_SLOT:
+	case XHCI_TRB_TYPE_RESET_DEVICE:
+		printf("slot %u c %u\n",
+			XHCI_TRB_3_SLOT_GET(trb_3),
+			_B(XHCI_TRB_3_CYCLE_BIT & trb_3));
+		break;
+	case XHCI_TRB_TYPE_ADDRESS_DEVICE:
+		printf("slot %u bsr %u ictx %016"PRIx64" c %u\n",
+			XHCI_TRB_3_SLOT_GET(trb_3),
+			_B(XHCI_TRB_3_BSR_BIT & trb_3),
+			trb_0,
+			_B(XHCI_TRB_3_CYCLE_BIT & trb_3));
+		break;
+	case XHCI_TRB_TYPE_CONFIGURE_EP:
+		printf("slot %u dc %u ictx %016"PRIx64" c %u\n",
+			XHCI_TRB_3_SLOT_GET(trb_3),
+			_B(XHCI_TRB_3_DCEP_BIT & trb_3),
+			trb_0,
+			_B(XHCI_TRB_3_CYCLE_BIT & trb_3));
+		break;
+	case XHCI_TRB_TYPE_EVALUATE_CTX:
+		printf("slot %u ictx %016"PRIx64" c %u\n",
+			XHCI_TRB_3_SLOT_GET(trb_3),
+			trb_0,
+			_B(XHCI_TRB_3_CYCLE_BIT & trb_3));
+		break;
+	case XHCI_TRB_TYPE_RESET_EP:
+		printf("slot %u ep %u prsv %u c %u\n",
+			XHCI_TRB_3_SLOT_GET(trb_3),
+			XHCI_TRB_3_EP_GET(trb_3),
+			_B(XHCI_TRB_3_PRSV_BIT & trb_3),
+			_B(XHCI_TRB_3_CYCLE_BIT & trb_3));
+		break;
+	case XHCI_TRB_TYPE_STOP_EP:
+		printf("slot %u ep %u sp %u c %u\n",
+			XHCI_TRB_3_SLOT_GET(trb_3),
+			XHCI_TRB_3_EP_GET(trb_3),
+			_B(XHCI_TRB_3_SUSP_EP_BIT & trb_3),
+			_B(XHCI_TRB_3_CYCLE_BIT & trb_3));
+		break;
+	case XHCI_TRB_TYPE_SET_TR_DEQUEUE:
+		printf("slot %u ep %u stream %u c %u ptr %016"PRIx64"\n",
+			XHCI_TRB_3_SLOT_GET(trb_3),
+			XHCI_TRB_3_EP_GET(trb_3),
+			XHCI_TRB_2_STREAM_GET(trb_2),
+			_B(XHCI_TRB_3_CYCLE_BIT & trb_3),
+			trb_0);
+		break;
+	case XHCI_TRB_TYPE_FORCE_EVENT:
+	case XHCI_TRB_TYPE_NEGOTIATE_BW:
+	case XHCI_TRB_TYPE_SET_LATENCY_TOL:
+	case XHCI_TRB_TYPE_GET_PORT_BW:
+	case XHCI_TRB_TYPE_FORCE_HEADER:
+	case XHCI_TRB_TYPE_NOOP_CMD:
+		printf("slot %u c %u\n",
+			XHCI_TRB_3_SLOT_GET(trb_3),
+			_B(XHCI_TRB_3_CYCLE_BIT & trb_3));
+		break;
+
+	case XHCI_TRB_EVENT_TRANSFER:
+		printf("%s(%u) slot %u ep %u ed %u c %u remlen %u ptr %016"
+			PRIx64"\n",
+			str_trberr(XHCI_TRB_2_ERROR_GET(trb_2)),
+			XHCI_TRB_2_ERROR_GET(trb_2),
+			XHCI_TRB_3_SLOT_GET(trb_3),
+			XHCI_TRB_3_EP_GET(trb_3),
+			_B(XHCI_TRB_3_ED_BIT & trb_3),
+			_B(XHCI_TRB_3_CYCLE_BIT & trb_3),
+			XHCI_TRB_2_REM_GET(trb_2),
+			trb_0);
+		break;
+	case XHCI_TRB_EVENT_CMD_COMPLETE:
+		printf("%s(%u) slot %u vf %u c %u param %u cmd %016"
+			PRIx64"\n",
+			str_trberr(XHCI_TRB_2_ERROR_GET(trb_2)),
+			XHCI_TRB_2_ERROR_GET(trb_2),
+			XHCI_TRB_3_SLOT_GET(trb_3),
+			XHCI_TRB_3_VFID_GET(trb_3),
+			_B(XHCI_TRB_3_CYCLE_BIT & trb_3),
+			XHCI_TRB_2_REM_GET(trb_2),
+			trb_0);
+		break;
+	case XHCI_TRB_EVENT_PORT_STS_CHANGE:
+		printf("%s(%u) port %u c %u\n",
+			str_trberr(XHCI_TRB_2_ERROR_GET(trb_2)),
+			XHCI_TRB_2_ERROR_GET(trb_2),
+			(uint32_t)((trb_0 >> 24) & 0xff),
+			_B(XHCI_TRB_3_CYCLE_BIT & trb_3));
+		break;
+	case XHCI_TRB_EVENT_HOST_CTRL:
+	case XHCI_TRB_EVENT_BW_REQUEST:
+	case XHCI_TRB_EVENT_DOORBELL:
+	case XHCI_TRB_EVENT_DEVICE_NOTIFY:
+	case XHCI_TRB_EVENT_MFINDEX_WRAP:
+		printf("%s(%u) c %u\n",
+			str_trberr(XHCI_TRB_2_ERROR_GET(trb_2)),
+			XHCI_TRB_2_ERROR_GET(trb_2),
+			_B(XHCI_TRB_3_CYCLE_BIT & trb_3));
+		break;
+	default:
+		printf("\n");
+		break;
+	};
+}
+
+static void __used
+xhci_dump_trb(int dbglvl, const struct xhci_trb * const trb)
+{
+	xhci_dump_trbn(dbglvl, trb->trb_0, trb->trb_2, trb->trb_3);
+}
+
+/* from umassvar.h */
+#define UR_BBB_GET_MAX_LUN	0xfe
+#define UR_BBB_RESET	0xff		/* Bulk-Only reset */
+
+static struct tbl_t reqtbl[] = {
+	TBLELEM(UR_GET_STATUS),
+	TBLELEM(UR_CLEAR_FEATURE),
+	TBLELEM(UR_SET_FEATURE),
+	TBLELEM(UR_SET_ADDRESS),
+	TBLELEM(UR_GET_DESCRIPTOR),
+	TBLELEM(UR_SET_DESCRIPTOR),
+	TBLELEM(UR_GET_CONFIG),
+	TBLELEM(UR_SET_CONFIG),
+	TBLELEM(UR_GET_INTERFACE),
+	TBLELEM(UR_SET_INTERFACE),
+	TBLELEM(UR_SYNCH_FRAME),
+	TBLELEM(UR_SET_ENCRYPTION),
+	TBLELEM(UR_GET_ENCRYPTION),
+	TBLELEM(UR_SET_HANDSHAKE),
+	TBLELEM(UR_GET_HANDSHAKE),
+	TBLELEM(UR_SET_CONNECTION),
+	TBLELEM(UR_SET_SECURITY_DATA),
+	TBLELEM(UR_GET_SECURITY_DATA),
+	TBLELEM(UR_SET_WUSB_DATA),
+	TBLELEM(UR_LOOPBACK_DATA_WRITE),
+	TBLELEM(UR_LOOPBACK_DATA_READ),
+	TBLELEM(UR_SET_INTERFACE_DS),
+	TBLELEM(UR_SET_SEL),
+	TBLELEM(UR_SET_ISOCH_DELAY),
+	TBLELEM(UR_BBB_GET_MAX_LUN),
+	TBLELEM(UR_BBB_RESET)
+};
+
+static struct tbl_t desctbl[] = {
+	TBLELEM(UDESC_DEVICE),
+	TBLELEM(UDESC_CONFIG),
+	TBLELEM(UDESC_STRING),
+	TBLELEM(UDESC_INTERFACE),
+	TBLELEM(UDESC_ENDPOINT),
+	TBLELEM(UDESC_DEVICE_QUALIFIER),
+	TBLELEM(UDESC_OTHER_SPEED_CONFIGURATION),
+	TBLELEM(UDESC_INTERFACE_POWER),
+	TBLELEM(UDESC_OTG),
+	TBLELEM(UDESC_DEBUG),
+	TBLELEM(UDESC_INTERFACE_ASSOC),
+	TBLELEM(UDESC_BOS),
+	TBLELEM(UDESC_DEVICE_CAPABILITY),
+	TBLELEM(UDESC_CS_DEVICE),
+	TBLELEM(UDESC_CS_CONFIG),
+	TBLELEM(UDESC_CS_STRING),
+	TBLELEM(UDESC_CS_INTERFACE),
+	TBLELEM(UDESC_CS_ENDPOINT),
+	TBLELEM(UDESC_HUB),
+	TBLELEM(UDESC_SSHUB),
+	TBLELEM(UDESC_SSEP_COMPANION),
+	TBLELEM(UDESC_SSP_ISOCHEP_COMPANION)
+};
+
+static struct tbl_t uftbl[] = {
+	TBLELEM(UF_ENDPOINT_HALT),
+	TBLELEM(UF_INTERFACE_FUNCTION_SUSPEND),
+	TBLELEM(UF_DEVICE_REMOTE_WAKEUP),
+	TBLELEM(UF_TEST_MODE),
+	TBLELEM(UF_DEVICE_B_HNP_ENABLE),
+	TBLELEM(UF_DEVICE_A_HNP_SUPPORT),
+	TBLELEM(UF_DEVICE_A_ALT_HNP_SUPPORT),
+	TBLELEM(UF_DEVICE_WUSB_DEVICE),
+	TBLELEM(UF_DEVICE_U1_ENABLE),
+	TBLELEM(UF_DEVICE_U2_ENABLE),
+	TBLELEM(UF_DEVICE_LTM_ENABLE),
+	TBLELEM(UF_DEVICE_B3_NTF_HOST_SEL),
+	TBLELEM(UF_DEVICE_B3_RESP_ENABLE),
+	TBLELEM(UF_DEVICE_LDM_ENABLE)
+};
+
+static struct tbl_t uhrtbl[] = {
+	TBLELEM(UR_GET_BUS_STATE),
+	TBLELEM(UR_CLEAR_TT_BUFFER),
+	TBLELEM(UR_RESET_TT),
+	TBLELEM(UR_GET_TT_STATE),
+	TBLELEM(UR_STOP_TT)
+};
+
+static struct tbl_t uhftbl[] = {
+//	TBLELEM(UHF_C_HUB_LOCAL_POWER),
+//	TBLELEM(UHF_C_HUB_OVER_CURRENT),
+	TBLELEM(UHF_PORT_CONNECTION),
+	TBLELEM(UHF_PORT_ENABLE),
+	TBLELEM(UHF_PORT_SUSPEND),
+	TBLELEM(UHF_PORT_OVER_CURRENT),
+	TBLELEM(UHF_PORT_RESET),
+	TBLELEM(UHF_PORT_POWER),
+	TBLELEM(UHF_PORT_LOW_SPEED),
+	TBLELEM(UHF_PORT_L1),
+	TBLELEM(UHF_C_PORT_CONNECTION),
+	TBLELEM(UHF_C_PORT_ENABLE),
+	TBLELEM(UHF_C_PORT_SUSPEND),
+	TBLELEM(UHF_C_PORT_OVER_CURRENT),
+	TBLELEM(UHF_C_PORT_RESET),
+	TBLELEM(UHF_PORT_TEST),
+	TBLELEM(UHF_PORT_INDICATOR),
+	TBLELEM(UHF_C_PORT_L1)
+};
+
+static const char *
+str_reqtype(int key)
+{
+	TBLLOOKUP(reqtbl, 3);
+}
+
+static const char *
+str_desctype(int key)
+{
+	TBLLOOKUP(desctbl, 6);
+}
+
+static const char *
+str_uftype(int key)
+{
+	TBLLOOKUP(uftbl, 3);
+}
+
+static const char * __used
+str_uhrtype(int key)
+{
+	TBLLOOKUP(uhrtbl, 3);
+}
+
+static const char *
+str_uhftype(int key)
+{
+	TBLLOOKUP(uhftbl, 4);
+}
+
+static void __used
+xhci_dump_desc(int dbglvl, void *p)
+{
+	usb_device_descriptor_t *ud = p;
+
+	switch (ud->bDescriptorType) {
+	case UDESC_DEVICE:
+		printf("%04x class %u subc %u proto %u mps %u noconf %u",
+			UGETW(ud->bcdUSB),
+			ud->bDeviceClass,
+			ud->bDeviceSubClass,
+			ud->bDeviceProtocol,
+			ud->bMaxPacketSize,
+			ud->bNumConfigurations);
+		break;
+	case UDESC_CONFIG: {
+		usb_config_descriptor_t *ucd = p;
+		printf("len %u noiface %u confval %u iconf %u attr %02x pwr %u",
+			UGETW(ucd->wTotalLength),
+			ucd->bNumInterface,
+			ucd->bConfigurationValue,
+			ucd->iConfiguration,
+			ucd->bmAttributes,
+			ucd->bMaxPower);
+		break;
+	}
+	case UDESC_STRING:
+	case UDESC_ENDPOINT:
+	case UDESC_BOS:
+		printf("notyet");
+		break;
+	case UDESC_HUB: {
+		usb_hub_descriptor_t *uhd = p;
+		printf("nports %u hubchar %04x pwrdelay %u cur %u",
+			uhd->bNbrPorts,
+			UGETW(uhd->wHubCharacteristics),
+			uhd->bPwrOn2PwrGood,
+			uhd->bHubContrCurrent);
+		break;
+	}
+	case UDESC_SSHUB:
+		printf("notyet");
+		break;
+	case UDESC_SSEP_COMPANION: {
+		usb_ssep_companion_descriptor_t *ussepd = p;
+		printf("maxburst %u attr %02x b/i %u",
+			ussepd->bMaxBurst,
+			ussepd->bmAttributes,
+			UGETW(ussepd->wBytesPerInterval));
+		break;
+	}
+	default:
+		break;
+	}
+}
+
+static void __used
+xhci_dump_reply(int dbglvl, usbd_xfer_handle xfer)
+{
+	usb_device_request_t *udr;
+
+	if (xhcidebug <= dbglvl)
+		return;
+
+	if (xfer == NULL)
+		return;
+
+	udr = &xfer->request;
+	if (udr->bRequest != UR_GET_DESCRIPTOR)
+		return;
+	switch (UGETW(udr->wValue) >> 8) {
+	case UDESC_CONFIG: /* conf, ifassoc, if, ep, ssep companion */
+	case UDESC_DEVICE:
+	case UDESC_BOS: /* bos, usb2 ext, ss dev cap, container id */
+	//case UDESC_STRING:
+	case UDESC_HUB:
+	case UDESC_SSHUB:
+		break;
+	default:
+		return;
+	}
+
+	int remlen = UGETW(udr->wLength);
+	uint8_t *ud = KERNADDR(&xfer->dmabuf, 0);
+	int comma = 0;
+	usb_device_descriptor_t *udd;
+	printf("getdesc: len %u: ", remlen);
+	while (remlen > 0) {
+		udd = (usb_device_descriptor_t *)ud;
+		if (udd->bLength == 0)
+			break;
+		printf("%s%s(%u) len %u<",
+			comma ? ">, " : "",
+			str_desctype(udd->bDescriptorType),
+			udd->bDescriptorType,
+			udd->bLength);
+		xhci_dump_desc(dbglvl, ud);
+		comma = 1;
+		remlen -= udd->bLength;
+		ud     += udd->bLength;
+	}
+	printf("%s\n", comma ? ">" : "");
+}
+
+static void __used
+xhci_dump_req(int dbglvl, void *p)
+{
+	usb_device_request_t *udr = p;
+	const char *type, *rcpt;
+
+	if (udr == NULL)
+		return;
+
+	if (xhcidebug <= dbglvl)
+		return;
+
+	switch (UT_GET_TYPE(udr->bmRequestType)) {
+	case UT_STANDARD:	type = "STAND";		break;
+	case UT_CLASS:		type = "CLASS";		break;
+	case UT_VENDOR:		type = "VENDOR";	break;
+	default:		type = "unknown";	break;
+	}
+	switch (UT_GET_RECIPIENT(udr->bmRequestType)) {
+	case UT_DEVICE:		rcpt = "DEV";		break;
+	case UT_INTERFACE:	rcpt = "IFACE";		break;
+	case UT_ENDPOINT:	rcpt = "ENDP";		break;
+	case UT_OTHER:		rcpt = "OTHER";		break;
+	default:		rcpt = "unknown";	break;
+	}
+	if (UT_GET_TYPE(udr->bmRequestType) != UT_STANDARD &&
+	    UT_GET_TYPE(udr->bmRequestType) != UT_CLASS) {
+		/* vendor specific request, no analysis any more */
+		printf("%02x %s,%s,%s val %04x idx %04x len %04x\n",
+			udr->bRequest,
+			UT_GET_DIR(udr->bmRequestType) == UT_WRITE ?
+			    "OUT" : "IN",
+			type, rcpt,
+			UGETW(udr->wValue),
+			UGETW(udr->wIndex),
+			UGETW(udr->wLength));
+		return;
+	}
+	printf("%s(%u) %s,%s,%s",
+		str_reqtype(udr->bRequest), udr->bRequest,
+		UT_GET_DIR(udr->bmRequestType) == UT_WRITE ? "OUT" : "IN",
+		type, rcpt);
+
+	switch (udr->bRequest) {
+	case UR_GET_DESCRIPTOR:
+		printf(" %s(0x%04x) idx %04x len %u\n",
+			str_desctype(UGETW(udr->wValue) >> 8),
+			UGETW(udr->wValue),
+			UGETW(udr->wIndex),
+			UGETW(udr->wLength));
+		break;
+	case UR_CLEAR_FEATURE:
+		if (UT_GET_RECIPIENT(udr->bmRequestType) == UT_OTHER) {
+			printf(" %s(%u) idx %u len %u\n",
+				str_uhftype(UGETW(udr->wValue) & 0xff),
+				UGETW(udr->wValue),
+				UGETW(udr->wIndex),
+				UGETW(udr->wLength));
+		} else {
+			printf(" %s(%u) idx %04x len %u\n",
+				str_uftype(UGETW(udr->wValue) & 0xff),
+				UGETW(udr->wValue),
+				UGETW(udr->wIndex),
+				UGETW(udr->wLength));
+		}
+		break;
+	case UR_SET_FEATURE:
+		if (UT_GET_RECIPIENT(udr->bmRequestType) == UT_OTHER) {
+			printf(" %s(%u) idx %u len %u\n",
+				str_uhftype(UGETW(udr->wValue) & 0xff),
+				UGETW(udr->wValue),
+				UGETW(udr->wIndex),
+				UGETW(udr->wLength));
+		} else {
+			printf(" %s(%u) idx %04x len %u\n",
+				str_uftype(UGETW(udr->wValue) & 0xff),
+				UGETW(udr->wValue),
+				UGETW(udr->wIndex),
+				UGETW(udr->wLength));
+		}
+		break;
+	case UR_SET_ADDRESS:
+	default:
+		printf(" val %04x idx %04x len %04x\n",
+			UGETW(udr->wValue),
+			UGETW(udr->wIndex),
+			UGETW(udr->wLength));
+		break;
+	}
+}
+
+
+static void __used
+xhci_dump_portsc(int dbglvl, uint32_t v)
+{
+	char sbuf[128];
+
+	if (xhcidebug <= dbglvl)
+		return;
+
+	snprintb(sbuf, sizeof sbuf, "\177\020"
+		"b\037WPR\0b\036DR\0b\033WOE\0b\032WDE\0b\031WCE\0"
+		"b\030CAS\0b\027CEC\0b\026PLC\0b\025PRC\0"
+		"b\024OCC\0b\023WRC\0b\022PEC\0b\021CSC\0b\020LWS\0"
+		"f\016\2PIC\0f\012\4SPEED\0b\011PP\0f\5\4PLS\0"
+		"b\4PR\0b\3OCA\0b\1PED\0b\0CCS\0\0",
+		v);
+	printf("%s\n", sbuf);
+}
+
+#undef TBLELEM
+#undef TBLLOOKUP
+#undef _B
+
+#endif /* XHCI_DEBUG */
--- xhcireg.h.orig	2014-09-09 11:24:29.000000000 +0900
+++ xhcireg.h	2014-09-16 11:02:28.000000000 +0900
 <at>  <at>  -266,6 +274,8  <at>  <at> 
 #define XHCI_TRB_3_FRID_SET(x)          (((x) & 0x7FF) << 20)
 #define XHCI_TRB_3_ISO_SIA_BIT          (1U << 31)
 #define XHCI_TRB_3_SUSP_EP_BIT          (1U << 23)
+#define XHCI_TRB_3_VFID_GET(x)          (((x) >> 16) & 0xFF)
+#define XHCI_TRB_3_VFID_SET(x)          (((x) & 0xFF) << 16)
 #define XHCI_TRB_3_SLOT_GET(x)          (((x) >> 24) & 0xFF)
 #define XHCI_TRB_3_SLOT_SET(x)          (((x) & 0xFF) << 24)

--- usb.h.orig	2014-09-09 11:25:20.000000000 +0900
+++ usb.h	2014-09-18 19:05:17.000000000 +0900
 <at>  <at>  -241,6 +165,8  <at>  <at> 
 #define  UDESC_OTG		0x09
 #define  UDESC_DEBUG		0x0a
 #define  UDESC_INTERFACE_ASSOC	0x0b
+#define  UDESC_BOS		0x0f
+#define  UDESC_DEVICE_CAPABILITY 0x10
 #define  UDESC_CS_DEVICE	0x21	/* class specific */
 #define  UDESC_CS_CONFIG	0x22
 #define  UDESC_CS_STRING	0x23
 <at>  <at>  -248,23 +174,47  <at>  <at> 
 #define  UDESC_CS_ENDPOINT	0x25
 #define  UDESC_HUB		0x29
 #define  UDESC_SSHUB		0x2a
+#define  UDESC_SSEP_COMPANION	0x30 /* SUPERSPEED_USB_ENDPOINT_COMPANION */
+#define  UDESC_SSP_ISOCHEP_COMPANION	0x31
+			/* SUPERSPEEDPLUS_ISOCHRONOUS_ENDPOINT_COMPANION */
 #define UR_SET_DESCRIPTOR	0x07
 #define UR_GET_CONFIG		0x08
 #define UR_SET_CONFIG		0x09
 #define UR_GET_INTERFACE	0x0a
 #define UR_SET_INTERFACE	0x0b
 #define UR_SYNCH_FRAME		0x0c
+#define UR_SET_ENCRYPTION	0x0d
+#define UR_GET_ENCRYPTION	0x0e
+#define UR_SET_HANDSHAKE	0x0f
+#define UR_GET_HANDSHAKE	0x10
+#define UR_SET_CONNECTION	0x11
+#define UR_SET_SECURITY_DATA	0x12
+#define UR_GET_SECURITY_DATA	0x13
+#define UR_SET_WUSB_DATA	0x14
+#define UR_LOOPBACK_DATA_WRITE	0x15
+#define UR_LOOPBACK_DATA_READ	0x16
+#define UR_SET_INTERFACE_DS	0x17
+#define UR_SET_SEL		0x30
+#define UR_SET_ISOCH_DELAY	0x31

 /*
  * Feature selectors. USB 2.0 spec, table 9-6 and OTG and EH suppliment,
  * table 6-2
  */
 #define UF_ENDPOINT_HALT	0
+#define UF_INTERFACE_FUNCTION_SUSPEND	0
 #define UF_DEVICE_REMOTE_WAKEUP	1
 #define UF_TEST_MODE		2
 #define UF_DEVICE_B_HNP_ENABLE	3
 #define UF_DEVICE_A_HNP_SUPPORT	4
 #define UF_DEVICE_A_ALT_HNP_SUPPORT 5
+#define UF_DEVICE_WUSB_DEVICE	6
+#define UF_DEVICE_U1_ENABLE	48
+#define UF_DEVICE_U2_ENABLE	49
+#define UF_DEVICE_LTM_ENABLE	50
+#define UF_DEVICE_B3_NTF_HOST_SEL	51
+#define UF_DEVICE_B3_RESP_ENABLE	52
+#define UF_DEVICE_LDM_ENABLE	53

 #define USB_MAX_IPACKET		8 /* maximum size of the initial packet */

 <at>  <at>  -371,6 +321,15  <at>  <at> 

 typedef struct {
 	uByte		bLength;
+	uByte		bDescriptorType;	/* 0x30 */
+	uByte		bMaxBurst;
+	uByte		bmAttributes;
+	uWord		wBytesPerInterval;
+} UPACKED usb_ssep_companion_descriptor_t;
+#define USB_SSEP_COMPANION_DESCRIPTOR_SIZE 6
+
+typedef struct {
+	uByte		bLength;
 	uByte		bDescriptorType;
 	uWord		bString[126];
 } UPACKED usb_string_descriptor_t;

Mark Davies | 19 Sep 06:29 2014
Picon
Picon

problems when multiple mfi cards in system?

I have a Dell PERC H810 pcie card and an external disk array attached 
to it.  If I put this into a desktop 6.1_STABLE/amd64 system I can 
happily read and write terabytes from the external array.  However if 
I move the card to a Dell poweredge r610 (which also has an internal 
PERC 6/i) and try reading from the external array, after a few minutes 
the entire machine just freezes solid.  There doesn't seem to be any 
problem reading from the internal disks.

Any ideas how to identify whats failing here and how to fix it?

dmesg for the r610 is below.

cheers
mark

Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 
2005,
    2006, 2007, 2008, 2009, 2010, 2011, 2012
    The NetBSD Foundation, Inc.  All rights reserved.
Copyright (c) 1982, 1986, 1989, 1991, 1993
    The Regents of the University of California.  All rights reserved.

NetBSD 6.1_STABLE (GENERIC) #25: Mon Jun  9 12:44:09 NZST 2014
	
mark <at> turakirae.ecs.vuw.ac.nz:/local/SAVE/6_64.obj/src/work/6/src/sys/arch/amd64/compile/GENERIC
total memory = 8182 MB
avail memory = 7929 MB
timecounter: Timecounters tick every 10.000 msec
timecounter: Timecounter "i8254" frequency 1193182 Hz quality 100
Dell Inc. PowerEdge R610
mainbus0 (root)
cpu0 at mainbus0 apid 16: Intel(R) Xeon(R) CPU           E5504   <at>  
2.00GHz, id 0x106a5
cpu1 at mainbus0 apid 18: Intel(R) Xeon(R) CPU           E5504   <at>  
2.00GHz, id 0x106a5
cpu2 at mainbus0 apid 20: Intel(R) Xeon(R) CPU           E5504   <at>  
2.00GHz, id 0x106a5
cpu3 at mainbus0 apid 22: Intel(R) Xeon(R) CPU           E5504   <at>  
2.00GHz, id 0x106a5
ioapic0 at mainbus0 apid 0: pa 0xfec00000, version 20, 24 pins
ioapic1 at mainbus0 apid 1: pa 0xfec80000, version 20, 24 pins
acpi0 at mainbus0: Intel ACPICA 20110623
acpi0: X/RSDT: OemId <DELL  ,PE_SC3  ,00000001>, AslId <DELL,00000001>
acpi0: SCI interrupting at int 9
timecounter: Timecounter "ACPI-Safe" frequency 3579545 Hz quality 900
hpet0 at acpi0: high precision event timer (mem 0xfed00000-0xfed00400)
timecounter: Timecounter "hpet0" frequency 14318180 Hz quality 2000
WHEA (PNP0C33) at acpi0 not configured
SPK (PNP0C01) at acpi0 not configured
attimer1 at acpi0 (TMR, PNP0100): io 0x40-0x5f irq 0
COMA (PNP0501) at acpi0 not configured
COMB (PNP0501) at acpi0 not configured
MBIO (PNP0C01) at acpi0 not configured
NIPM (IPI0001) at acpi0 not configured
MBI1 (PNP0C01) at acpi0 not configured
PEHB (PNP0C02) at acpi0 not configured
VTD (PNP0C02) at acpi0 not configured
ipmi0 at mainbus0
pci0 at mainbus0 bus 0: configuration mode 1
pci0: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
pchb0 at pci0 dev 0 function 0: vendor 0x8086 product 0x3403 (rev. 
0x13)
ppb0 at pci0 dev 1 function 0: vendor 0x8086 product 0x3408 (rev. 
0x13)
ppb0: PCI Express 2.0 <Root Port of PCI-E Root Complex>
pci1 at ppb0 bus 1
pci1: i/o space, memory space enabled, rd/line, wr/inv ok
bnx0 at pci1 dev 0 function 0: Broadcom NetXtreme II BCM5709 1000Base-
T
bnx0: Ethernet address 00:22:19:60:83:95
bnx0: interrupting at ioapic1 pin 4
brgphy0 at bnx0 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8
brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 
1000baseT-FDX, auto
bnx1 at pci1 dev 0 function 1: Broadcom NetXtreme II BCM5709 1000Base-
T
bnx1: Ethernet address 00:22:19:60:83:97
bnx1: interrupting at ioapic1 pin 16
brgphy1 at bnx1 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8
brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 
1000baseT-FDX, auto
ppb1 at pci0 dev 3 function 0: vendor 0x8086 product 0x340a (rev. 
0x13)
ppb1: PCI Express 2.0 <Root Port of PCI-E Root Complex>
pci2 at ppb1 bus 2
pci2: i/o space, memory space enabled, rd/line, wr/inv ok
bnx2 at pci2 dev 0 function 0: Broadcom NetXtreme II BCM5709 1000Base-
T
bnx2: Ethernet address 00:22:19:60:83:99
bnx2: interrupting at ioapic1 pin 0
brgphy2 at bnx2 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8
brgphy2: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 
1000baseT-FDX, auto
bnx3 at pci2 dev 0 function 1: Broadcom NetXtreme II BCM5709 1000Base-
T
bnx3: Ethernet address 00:22:19:60:83:9b
bnx3: interrupting at ioapic1 pin 10
brgphy3 at bnx3 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8
brgphy3: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 
1000baseT-FDX, auto
ppb2 at pci0 dev 7 function 0: vendor 0x8086 product 0x340e (rev. 
0x13)
ppb2: PCI Express 2.0 <Root Port of PCI-E Root Complex>
pci3 at ppb2 bus 4
pci3: i/o space, memory space enabled, rd/line, wr/inv ok
ppb3 at pci0 dev 9 function 0: vendor 0x8086 product 0x3410 (rev. 
0x13)
ppb3: PCI Express 2.0 <Root Port of PCI-E Root Complex>
pci4 at ppb3 bus 5
pci4: i/o space, memory space enabled, rd/line, wr/inv ok
mfi0 at pci4 dev 0 function 0: Dell PERC H810 Adapter
mfi0: interrupting at ioapic1 pin 8
mfi0: PERC H810 Adapter version 21.0.1-0132
mfi0: logical drives 1, 1024MB RAM, BBU type BBU, status good
scsibus0 at mfi0: 64 targets, 8 luns per target
vendor 0x8086 product 0x342e (interrupt system, revision 0x13) at pci0 
dev 20 function 0 not configured
vendor 0x8086 product 0x3422 (interrupt system, revision 0x13) at pci0 
dev 20 function 1 not configured
vendor 0x8086 product 0x3423 (interrupt system, revision 0x13) at pci0 
dev 20 function 2 not configured
uhci0 at pci0 dev 26 function 0: vendor 0x8086 product 0x2937 (rev. 
0x02)
uhci0: interrupting at ioapic0 pin 17
usb0 at uhci0: USB revision 1.0
uhci1 at pci0 dev 26 function 1: vendor 0x8086 product 0x2938 (rev. 
0x02)
uhci1: interrupting at ioapic0 pin 18
usb1 at uhci1: USB revision 1.0
ehci0 at pci0 dev 26 function 7: vendor 0x8086 product 0x293c (rev. 
0x02)
ehci0: interrupting at ioapic0 pin 19
ehci0: BIOS has given up ownership
ehci0: EHCI version 1.0
ehci0: companion controllers, 2 ports each: uhci0 uhci1
usb2 at ehci0: USB revision 2.0
ppb4 at pci0 dev 28 function 0: vendor 0x8086 product 0x2940 (rev. 
0x02)
ppb4: PCI Express 1.0 <Root Port of PCI-E Root Complex>
pci5 at ppb4 bus 3
pci5: i/o space, memory space enabled, rd/line, wr/inv ok
mfi1 at pci5 dev 0 function 0: Dell PERC 6/i integrated
mfi1: interrupting at ioapic0 pin 16
mfi1: PERC 6/i Integrated version 6.2.0-0013
mfi1: logical drives 1, 256MB RAM, BBU type BBU, status good
scsibus1 at mfi1: 64 targets, 8 luns per target
uhci2 at pci0 dev 29 function 0: vendor 0x8086 product 0x2934 (rev. 
0x02)
uhci2: interrupting at ioapic0 pin 21
usb3 at uhci2: USB revision 1.0
uhci3 at pci0 dev 29 function 1: vendor 0x8086 product 0x2935 (rev. 
0x02)
uhci3: interrupting at ioapic0 pin 20
usb4 at uhci3: USB revision 1.0
ehci1 at pci0 dev 29 function 7: vendor 0x8086 product 0x293a (rev. 
0x02)
ehci1: interrupting at ioapic0 pin 21
ehci1: EHCI version 1.0
ehci1: companion controllers, 2 ports each: uhci2 uhci3
usb5 at ehci1: USB revision 2.0
ppb5 at pci0 dev 30 function 0: vendor 0x8086 product 0x244e (rev. 
0x92)
pci6 at ppb5 bus 6
pci6: i/o space, memory space enabled
vga0 at pci6 dev 3 function 0: vendor 0x102b product 0x0532 (rev. 
0x0a)
wsdisplay0 at vga0 kbdmux 1: console (80x25, vt100 emulation)
wsmux1: connecting to wsdisplay0
drm at vga0 not configured
ichlpcib0 at pci0 dev 31 function 0: vendor 0x8086 product 0x2918 
(rev. 0x02)
timecounter: Timecounter "ichlpcib0" frequency 3579545 Hz quality 1000
ichlpcib0: 24-bit timer
ichlpcib0: TCO (watchdog) timer configured.
piixide0 at pci0 dev 31 function 2: Intel 82801I Serial ATA Controller 
(ICH9) (rev. 0x02)
piixide0: bus-master DMA support present
piixide0: primary channel configured to native-PCI mode
piixide0: using ioapic0 pin 23 for native-PCI interrupt
atabus0 at piixide0 channel 0
piixide0: secondary channel configured to native-PCI mode
atabus1 at piixide0 channel 1
isa0 at ichlpcib0
tpm0 at isa0 iomem 0xfed40000-0xfed44fff irq 7: device 0x4a100000 rev 
0x4e
com0 at isa0 port 0x3f8-0x3ff irq 4: ns16550a, working fifo
com1 at isa0 port 0x2f8-0x2ff irq 3: ns16550a, working fifo
pckbc0 at isa0 port 0x60-0x64
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
midi0 at pcppi0: PC speaker
sysbeep0 at pcppi0
attimer1: attached to pcppi0
acpicpu0 at cpu0: ACPI CPU
acpicpu0: C1: FFH, lat   1 us, pow  1000 mW
acpicpu0: C3: FFH, lat  96 us, pow   350 mW
acpicpu0: P0: FFH, lat  10 us, pow 80000 mW, 1995 MHz
acpicpu0: P1: FFH, lat  10 us, pow 66000 mW, 1862 MHz
acpicpu0: P2: FFH, lat  10 us, pow 55000 mW, 1729 MHz
acpicpu0: P3: FFH, lat  10 us, pow 45000 mW, 1596 MHz
coretemp0 at cpu0: thermal sensor, 1 C resolution
acpicpu1 at cpu1: ACPI CPU
coretemp1 at cpu1: thermal sensor, 1 C resolution
acpicpu2 at cpu2: ACPI CPU
coretemp2 at cpu2: thermal sensor, 1 C resolution
acpicpu3 at cpu3: ACPI CPU
coretemp3 at cpu3: thermal sensor, 1 C resolution
timecounter: Timecounter "clockinterrupt" frequency 100 Hz quality 0
timecounter: Timecounter "TSC" frequency 1995099040 Hz quality 3000
scsibus0: waiting 2 seconds for devices to settle...
scsibus1: waiting 2 seconds for devices to settle...
uhub0 at usb3: vendor 0x8086 UHCI root hub, class 9/0, rev 1.00/1.00, 
addr 1
uhub0: 2 ports with 2 removable, self powered
uhub1 at usb1: vendor 0x8086 UHCI root hub, class 9/0, rev 1.00/1.00, 
addr 1
uhub1: 2 ports with 2 removable, self powered
uhub2 at usb0: vendor 0x8086 UHCI root hub, class 9/0, rev 1.00/1.00, 
addr 1
uhub2: 2 ports with 2 removable, self powered
uhub3 at usb5: vendor 0x8086 EHCI root hub, class 9/0, rev 2.00/1.00, 
addr 1
uhub3: 4 ports with 4 removable, self powered
uhub4 at usb4: vendor 0x8086 UHCI root hub, class 9/0, rev 1.00/1.00, 
addr 1
uhub4: 2 ports with 2 removable, self powered
uhub5 at usb2: vendor 0x8086 EHCI root hub, class 9/0, rev 2.00/1.00, 
addr 1
uhub5: 4 ports with 4 removable, self powered
sd0 at scsibus0 target 0 lun 0: <DELL, PERC H810, 3.13> disk fixed
sd0: fabricating a geometry
sd0: 8382 GB, 8583168 cyl, 64 head, 32 sec, 512 bytes/sect x 
17578328064 sectors
sd0: fabricating a geometry
sd0: GPT GUID: e8829692-32f1-11e4-863d-180373e73b4d
dk0 at sd0: e8829699-32f1-11e4-863d-180373e73b4d
dk0: 17578327967 blocks at 64, type: ffs
sd0: tagged queueing
sd1 at scsibus1 target 0 lun 0: <DELL, PERC 6/i, 1.22> disk fixed
sd1: fabricating a geometry
sd1: 557 GB, 571136 cyl, 64 head, 32 sec, 512 bytes/sect x 1169686528 
sectors
sd1: fabricating a geometry
sd1: tagged queueing
uhub6 at uhub5 port 3: vendor 0x0424 product 0x2514, class 9/0, rev 
2.00/0.00, addr 2
uhub6: multiple transaction translators
uhub6: 3 ports with 3 removable, self powered
uhidev0 at uhub2 port 1 configuration 1 interface 0
uhidev0: Avocent Dell 03R874, rev 1.10/1.00, addr 2, iclass 3/1
ukbd0 at uhidev0
uhidev1 at uhub0 port 2 configuration 1 interface 0
uhidev1: Avocent USB Composite Device-0, rev 1.10/0.00, addr 2, iclass 
3/1
ukbd1 at uhidev1
wskbd1 at ukbd0 mux 1
wskbd1: connecting to wsdisplay0
uhidev2 at uhub2 port 1 configuration 1 interface 1
uhidev2: Avocent Dell 03R874, rev 1.10/1.00, addr 2, iclass 3/1
uhidev2: 3 report ids
ums0 at uhidev2 reportid 1: 5 buttons and Z dir
wsmouse0 at ums0 mux 0
uhid0 at uhidev2 reportid 2: input=2, output=0, feature=0
uhid1 at uhidev2 reportid 3: input=1, output=0, feature=0
wskbd2 at ukbd1 mux 1
wskbd2: connecting to wsdisplay0
uhidev3 at uhub0 port 2 configuration 1 interface 1
uhidev3: Avocent USB Composite Device-0, rev 1.10/0.00, addr 2, iclass 
3/1
ums1 at uhidev3: 3 buttons and Z dir
wsmouse1 at ums1 mux 0
atapibus0 at atabus0: 2 targets
cd0 at atapibus0 drive 0: <TSSTcorp DVD+/-RW TS-L633C, 
l1M3456789KLMNOP, D150> cdrom removable
cd0: 32-bit data port
cd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 5 
(Ultra/100)
cd0(piixide0:0:0): using PIO mode 4, Ultra-DMA mode 5 (Ultra/100) 
(using DMA)
Kernelized RAIDframe activated
pad0: outputs: 44100Hz, 16-bit, stereo
audio0 at pad0: half duplex, playback, capture
boot device: sd1
root on sd1a dumps on sd1b
root file system type: ffs
/var: replaying log to disk
/usr: replaying log to disk
mfi1: normal state on 'mfi1:0' (online)
mfi0: normal state on 'mfi0:0' (online)
wsdisplay0: screen 1 added (80x25, vt100 emulation)
wsdisplay0: screen 2 added (80x25, vt100 emulation)
wsdisplay0: screen 3 added (80x25, vt100 emulation)
wsdisplay0: screen 4 added (80x25, vt100 emulation)
ipmi0: version 2.0 interface KCS iobase 0xca8/8 spacing 4
/local: replaying log to disk
/am/state-opera/home1: replaying log to disk
/am/state-opera/x-home: replaying log to disk
/am/state-opera/vol/scratch: replaying log to disk
/scratch: replaying log to disk
/backup/X: replaying log to disk
Accounting started


Gmane