headers
Paul Robertson | 13 Jul 2011 18:48
Picon

"vxplex dis" error "v-5-1-2321 cannot lock into memory:resource temporarily unavailable" with SFORAC 5.0MP3

We are currently running SFORA RAC 5.0MP3 on Solaris 10 64-bit two-node cluster. We run a large 10TB
Oracle RAC database on this cluster, and we are in the process of VxVM mirroring the data from an
old EMC array to a new EMC array.

We provisioned the new EMC storage without incident, added the new disks to the existing diskgroups,
and the mirrored the data between arrays. As the final step in the process, we then ran "vxplex -g
dgname dis plex-nn", and saw the following for three separate volumes:


v  arch         -            ENABLED  ACTIVE   6487707904 SELECT  -       fsgen
pl arch-01      arch         ENABLED  ACTIVE   6487707904 CONCAT  -       RW
sd 0804088C-01  arch-01      0804088C 0        1696907008 0               c3t50000972C00C915Cd25 ENA
sd 080408BC-01  arch-01      080408BC 0        1696907008 1696907008      c3t50000972C00C915Cd26 ENA
sd 080408EC-01  arch-01      080408EC 0        1696907008 3393814016      c3t50000972C00C915Cd27 ENA
sd 0804091C-01  arch-01      0804091C 0        1396986880 5090721024      c3t50000972C00C915Cd28 ENA
pl arch-02      arch         ENABLED  ACTIVE   6487707904 CONCAT  -       RW
sd 20520113-01  arch-02      20520113 0        1428029344 0               c3t5006016844600097d8 ENA
sd 20520061-02  arch-02      20520061 861984256 666538784 1428029344      c2t5006016244600097d12 ENA
sd 20520121-01  arch-02      20520121 0        1537081088 2094568128      c2t5006016244600097d16 ENA
sd 20520064-01  arch-02      20520064 0        1428029344 3631649216      c3t5006016844600097d25 ENA
sd 20520103-01  arch-02      20520103 0        1428029344 5059678560      c3t5006016844600097d26 ENA

racnyc05:~>sudo vxplex -g oradg dis arch-02
VxVM vxsync ERROR V-5-1-2321 Cannot lock into memory: Resource temporarily unavailable
VxVM vxplex ERROR V-5-1-10870 fsgen/vxplex: Warning: vxsync exited with exitcode 42:
        Volume data may not be flushed to all plexes

Despite the warning above, it seemed to work, however, in that the
plex is disassociated and the filesystem is still accessible:

racnyc05:~>vxprint -htg oradg arch arch-02
V  NAME         RVG/VSET/CO  KSTATE   STATE    LENGTH   READPOL   PREFPLEX UTYPE
PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE
SC NAME         PLEX         CACHE    DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
DC NAME         PARENTVOL    LOGVOL
SP NAME         SNAPVOL      DCO
EX NAME         ASSOC        VC                       PERMS    MODE     STATE
SR NAME         KSTATE

pl arch-02      -            DISABLED -        6487707904 CONCAT  -        RW
sd 20520113-01  arch-02      20520113 0        1428029344 0       c3t5006016844600097d8 ENA
sd 20520061-02  arch-02      20520061 861984256 666538784 1428029344 c2t5006016244600097d12 ENA
sd 20520121-01  arch-02      20520121 0        1537081088 2094568128 c2t5006016244600097d16 ENA
sd 20520064-01  arch-02      20520064 0        1428029344 3631649216 c3t5006016844600097d25 ENA
sd 20520103-01  arch-02      20520103 0        1428029344 5059678560 c3t5006016844600097d26 ENA

v  arch         -            ENABLED  ACTIVE   6487707904 SELECT  -        fsgen
pl arch-01      arch         ENABLED  ACTIVE   6487707904 CONCAT  -        RW
sd 0804088C-01  arch-01      0804088C 0        1696907008 0       c3t50000972C00C915Cd25 ENA
sd 080408BC-01  arch-01      080408BC 0        1696907008 1696907008 c3t50000972C00C915Cd26 ENA
sd 080408EC-01  arch-01      080408EC 0        1696907008 3393814016 c3t50000972C00C915Cd27 ENA
sd 0804091C-01  arch-01      0804091C 0        1396986880 5090721024 c3t50000972C00C915Cd28 ENA

racnyc05:~>df -h /arch
Filesystem             size   used  avail capacity  Mounted on
/dev/vx/dsk/oradg/arch
                       3.0T    54G   2.8T     2%    /arch

racnyc05:~>sudo fstyp /dev/vx/rdsk/oradg/arch vxfs

It's happened on three volumes in two different clustered diskgroups
on this cluster. I'm unwilling to proceed with the more critical
volumes until I get a better understanding of the root cause. I'm
concerned that the remaining active plex on the new array may become
inconsistent during the plex disassociation, which could cause serious
problems for our database.

Here's a little more information:

racnyc05:~>modinfo | grep -i vx
 39 7bea2000  3e4e0 307   1  vxdmp (VxVM 5.0MP3: DMP Driver)
 41 7ba00000 209248 272   1  vxio (VxVM 5.0MP3 I/O driver)
 43 7bea11e8    c78 273   1  vxspec (VxVM 5.0MP3 control/status driv)
236 7afc54a8    cb0 310   1  vxportal (VxFS 5.0_REV-5.0MP3A25_sol port)
240 7a600000 1d89e0  20   1  vxfs (VxFS 5.0_REV-5.0MP3A25_sol SunO)
260 7a7ec000   a9e0 311   1  fdd (VxQIO 5.0_REV-5.0MP3A25_sol Qui)
264 7ab4a000  51c10 315   1  vxfen (VRTS Fence 5.0MP3)
265 7b600000  21ec0 316   1  vxglm (VxGLM 5.0MP3 (SunOS 5.10))
266 7ab9e000   5418 317   1  vxgms (VxGMS 5.0MP3 (SunOS))


Let me know if you need any additional information.

Cheers,

Paul

<div><p>We are currently running SFORA RAC 5.0MP3 on Solaris 10 64-bit two-node cluster. We run a large 10TB<br>Oracle RAC database on this cluster, and we are in the process of VxVM mirroring the data from an<br>old EMC array to a new EMC array.<br><br>We provisioned the new EMC storage without incident, added the new disks to the existing diskgroups,<br>and the mirrored the data between arrays. As the final step in the process, we then ran "vxplex -g<br>dgname dis plex-nn", and saw the following for three separate volumes:<br><br><br>v&nbsp; arch&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ENABLED&nbsp; ACTIVE&nbsp;&nbsp; 6487707904 SELECT&nbsp; -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; fsgen<br>pl arch-01&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; arch&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ENABLED&nbsp; ACTIVE&nbsp;&nbsp; 6487707904 CONCAT&nbsp; -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; RW<br>sd 0804088C-01&nbsp; arch-01&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0804088C 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1696907008 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; c3t50000972C00C915Cd25 ENA<br>
sd 080408BC-01&nbsp; arch-01&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 080408BC 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1696907008 1696907008&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; c3t50000972C00C915Cd26 ENA<br>sd 080408EC-01&nbsp; arch-01&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 080408EC 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1696907008 3393814016&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; c3t50000972C00C915Cd27 ENA<br>sd 0804091C-01&nbsp; arch-01&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0804091C 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1396986880 5090721024&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; c3t50000972C00C915Cd28 ENA<br>
pl arch-02&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; arch&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ENABLED&nbsp; ACTIVE&nbsp;&nbsp; 6487707904 CONCAT&nbsp; -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; RW<br>sd 20520113-01&nbsp; arch-02&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 20520113 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1428029344 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; c3t5006016844600097d8 ENA<br>sd 20520061-02&nbsp; arch-02&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 20520061 861984256 666538784 1428029344&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; c2t5006016244600097d12 ENA<br>
sd 20520121-01&nbsp; arch-02&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 20520121 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1537081088 2094568128&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; c2t5006016244600097d16 ENA<br>sd 20520064-01&nbsp; arch-02&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 20520064 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1428029344 3631649216&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; c3t5006016844600097d25 ENA<br>sd 20520103-01&nbsp; arch-02&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 20520103 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1428029344 5059678560&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; c3t5006016844600097d26 ENA<br><br>racnyc05:~&gt;sudo vxplex -g oradg dis arch-02<br>VxVM vxsync ERROR V-5-1-2321 Cannot lock into memory: Resource temporarily unavailable<br>VxVM vxplex ERROR V-5-1-10870 fsgen/vxplex: Warning: vxsync exited with exitcode 42:<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Volume data may not be flushed to all plexes<br><br>Despite the warning above, it seemed to work, however, in that the<br>plex is disassociated and the filesystem is still accessible:<br><br>racnyc05:~&gt;vxprint -htg oradg arch arch-02<br>
V&nbsp; NAME&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; RVG/VSET/CO&nbsp; KSTATE&nbsp;&nbsp; STATE&nbsp;&nbsp;&nbsp; LENGTH&nbsp;&nbsp; READPOL&nbsp;&nbsp; PREFPLEX UTYPE<br>PL NAME&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; VOLUME&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; KSTATE&nbsp;&nbsp; STATE&nbsp;&nbsp;&nbsp; LENGTH&nbsp;&nbsp; LAYOUT&nbsp;&nbsp;&nbsp; NCOL/WID MODE<br>SD NAME&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; PLEX&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; DISK&nbsp;&nbsp;&nbsp;&nbsp; DISKOFFS LENGTH&nbsp;&nbsp; [COL/]OFF DEVICE&nbsp;&nbsp; MODE<br>
SV NAME&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; PLEX&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; VOLNAME&nbsp; NVOLLAYR LENGTH&nbsp;&nbsp; [COL/]OFF AM/NM&nbsp;&nbsp;&nbsp; MODE<br>SC NAME&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; PLEX&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; CACHE&nbsp;&nbsp;&nbsp; DISKOFFS LENGTH&nbsp;&nbsp; [COL/]OFF DEVICE&nbsp;&nbsp; MODE<br>DC NAME&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; PARENTVOL&nbsp;&nbsp;&nbsp; LOGVOL<br>SP NAME&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; SNAPVOL&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; DCO<br>
EX NAME&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ASSOC&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; VC&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; PERMS&nbsp;&nbsp;&nbsp; MODE&nbsp;&nbsp;&nbsp;&nbsp; STATE<br>SR NAME&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; KSTATE<br><br>pl arch-02&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; DISABLED -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 6487707904 CONCAT&nbsp; -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; RW<br>sd 20520113-01&nbsp; arch-02&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 20520113 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1428029344 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; c3t5006016844600097d8 ENA<br>
sd 20520061-02&nbsp; arch-02&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 20520061 861984256 666538784 1428029344 c2t5006016244600097d12 ENA<br>sd 20520121-01&nbsp; arch-02&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 20520121 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1537081088 2094568128 c2t5006016244600097d16 ENA<br>sd 20520064-01&nbsp; arch-02&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 20520064 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1428029344 3631649216 c3t5006016844600097d25 ENA<br>
sd 20520103-01&nbsp; arch-02&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 20520103 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1428029344 5059678560 c3t5006016844600097d26 ENA<br><br>v&nbsp; arch&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ENABLED&nbsp; ACTIVE&nbsp;&nbsp; 6487707904 SELECT&nbsp; -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; fsgen<br>pl arch-01&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; arch&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ENABLED&nbsp; ACTIVE&nbsp;&nbsp; 6487707904 CONCAT&nbsp; -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; RW<br>
sd 0804088C-01&nbsp; arch-01&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0804088C 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1696907008 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; c3t50000972C00C915Cd25 ENA<br>sd 080408BC-01&nbsp; arch-01&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 080408BC 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1696907008 1696907008 c3t50000972C00C915Cd26 ENA<br>sd 080408EC-01&nbsp; arch-01&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 080408EC 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1696907008 3393814016 c3t50000972C00C915Cd27 ENA<br>
sd 0804091C-01&nbsp; arch-01&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0804091C 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1396986880 5090721024 c3t50000972C00C915Cd28 ENA<br><br>racnyc05:~&gt;df -h /arch<br>Filesystem&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; size&nbsp;&nbsp; used&nbsp; avail capacity&nbsp; Mounted on<br>/dev/vx/dsk/oradg/arch<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 3.0T&nbsp;&nbsp;&nbsp; 54G&nbsp;&nbsp; 2.8T&nbsp;&nbsp;&nbsp;&nbsp; 2%&nbsp;&nbsp;&nbsp; /arch<br><br>racnyc05:~&gt;sudo fstyp /dev/vx/rdsk/oradg/arch vxfs<br><br>It's happened on three volumes in two different clustered diskgroups<br>on this cluster. I'm unwilling to proceed with the more critical<br>
volumes until I get a better understanding of the root cause. I'm<br>concerned that the remaining active plex on the new array may become<br>inconsistent during the plex disassociation, which could cause serious<br>problems for our database.<br><br>Here's a little more information:<br><br>racnyc05:~&gt;modinfo | grep -i vx<br>&nbsp;39 7bea2000&nbsp; 3e4e0 307&nbsp;&nbsp; 1&nbsp; vxdmp (VxVM 5.0MP3: DMP Driver)<br>&nbsp;41 7ba00000 209248 272&nbsp;&nbsp; 1&nbsp; vxio (VxVM 5.0MP3 I/O driver)<br>&nbsp;43 7bea11e8&nbsp;&nbsp;&nbsp; c78 273&nbsp;&nbsp; 1&nbsp; vxspec (VxVM 5.0MP3 control/status driv)<br>
236 7afc54a8&nbsp;&nbsp;&nbsp; cb0 310&nbsp;&nbsp; 1&nbsp; vxportal (VxFS 5.0_REV-5.0MP3A25_sol port)<br>240 7a600000 1d89e0&nbsp; 20&nbsp;&nbsp; 1&nbsp; vxfs (VxFS 5.0_REV-5.0MP3A25_sol SunO)<br>260 7a7ec000&nbsp;&nbsp; a9e0 311&nbsp;&nbsp; 1&nbsp; fdd (VxQIO 5.0_REV-5.0MP3A25_sol Qui)<br>264 7ab4a000&nbsp; 51c10 315&nbsp;&nbsp; 1&nbsp; vxfen (VRTS Fence 5.0MP3)<br>
265 7b600000&nbsp; 21ec0 316&nbsp;&nbsp; 1&nbsp; vxglm (VxGLM 5.0MP3 (SunOS 5.10))<br>266 7ab9e000&nbsp;&nbsp; 5418 317&nbsp;&nbsp; 1&nbsp; vxgms (VxGMS 5.0MP3 (SunOS))<br><br><br>Let me know if you need any additional information.<br><br>Cheers,<br><br>Paul<br></p></div>
Permalink | Reply | 4 comments |
headers
William Havey | 13 Jul 2011 21:22
Picon

Re: "vxplex dis" error "v-5-1-2321 cannot lock into memory:resource temporarily unavailable" with SFORAC 5.0MP3

Paul,
 
Before you disassociate the plex, did you run a vxtask list|monitor to determine that the mirroring process has actually completed?
 
Bill

On Wed, Jul 13, 2011 at 12:48 PM, Paul Robertson <paul.marc.robertson <at> gmail.com> wrote:
We are currently running SFORA RAC 5.0MP3 on Solaris 10 64-bit two-node cluster. We run a large 10TB
Oracle RAC database on this cluster, and we are in the process of VxVM mirroring the data from an
old EMC array to a new EMC array.

We provisioned the new EMC storage without incident, added the new disks to the existing diskgroups,
and the mirrored the data between arrays. As the final step in the process, we then ran "vxplex -g
dgname dis plex-nn", and saw the following for three separate volumes:


v  arch         -            ENABLED  ACTIVE   6487707904 SELECT  -       fsgen
pl arch-01      arch         ENABLED  ACTIVE   6487707904 CONCAT  -       RW
sd 0804088C-01  arch-01      0804088C 0        1696907008 0               c3t50000972C00C915Cd25 ENA
sd 080408BC-01  arch-01      080408BC 0        1696907008 1696907008      c3t50000972C00C915Cd26 ENA
sd 080408EC-01  arch-01      080408EC 0        1696907008 3393814016      c3t50000972C00C915Cd27 ENA
sd 0804091C-01  arch-01      0804091C 0        1396986880 5090721024      c3t50000972C00C915Cd28 ENA
pl arch-02      arch         ENABLED  ACTIVE   6487707904 CONCAT  -       RW
sd 20520113-01  arch-02      20520113 0        1428029344 0               c3t5006016844600097d8 ENA
sd 20520061-02  arch-02      20520061 861984256 666538784 1428029344      c2t5006016244600097d12 ENA
sd 20520121-01  arch-02      20520121 0        1537081088 2094568128      c2t5006016244600097d16 ENA
sd 20520064-01  arch-02      20520064 0        1428029344 3631649216      c3t5006016844600097d25 ENA
sd 20520103-01  arch-02      20520103 0        1428029344 5059678560      c3t5006016844600097d26 ENA

racnyc05:~>sudo vxplex -g oradg dis arch-02
VxVM vxsync ERROR V-5-1-2321 Cannot lock into memory: Resource temporarily unavailable
VxVM vxplex ERROR V-5-1-10870 fsgen/vxplex: Warning: vxsync exited with exitcode 42:
        Volume data may not be flushed to all plexes

Despite the warning above, it seemed to work, however, in that the
plex is disassociated and the filesystem is still accessible:

racnyc05:~>vxprint -htg oradg arch arch-02
V  NAME         RVG/VSET/CO  KSTATE   STATE    LENGTH   READPOL   PREFPLEX UTYPE
PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE
SC NAME         PLEX         CACHE    DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
DC NAME         PARENTVOL    LOGVOL
SP NAME         SNAPVOL      DCO
EX NAME         ASSOC        VC                       PERMS    MODE     STATE
SR NAME         KSTATE

pl arch-02      -            DISABLED -        6487707904 CONCAT  -        RW
sd 20520113-01  arch-02      20520113 0        1428029344 0       c3t5006016844600097d8 ENA
sd 20520061-02  arch-02      20520061 861984256 666538784 1428029344 c2t5006016244600097d12 ENA
sd 20520121-01  arch-02      20520121 0        1537081088 2094568128 c2t5006016244600097d16 ENA
sd 20520064-01  arch-02      20520064 0        1428029344 3631649216 c3t5006016844600097d25 ENA
sd 20520103-01  arch-02      20520103 0        1428029344 5059678560 c3t5006016844600097d26 ENA

v  arch         -            ENABLED  ACTIVE   6487707904 SELECT  -        fsgen
pl arch-01      arch         ENABLED  ACTIVE   6487707904 CONCAT  -        RW
sd 0804088C-01  arch-01      0804088C 0        1696907008 0       c3t50000972C00C915Cd25 ENA
sd 080408BC-01  arch-01      080408BC 0        1696907008 1696907008 c3t50000972C00C915Cd26 ENA
sd 080408EC-01  arch-01      080408EC 0        1696907008 3393814016 c3t50000972C00C915Cd27 ENA
sd 0804091C-01  arch-01      0804091C 0        1396986880 5090721024 c3t50000972C00C915Cd28 ENA

racnyc05:~>df -h /arch
Filesystem             size   used  avail capacity  Mounted on
/dev/vx/dsk/oradg/arch
                       3.0T    54G   2.8T     2%    /arch

racnyc05:~>sudo fstyp /dev/vx/rdsk/oradg/arch vxfs

It's happened on three volumes in two different clustered diskgroups
on this cluster. I'm unwilling to proceed with the more critical
volumes until I get a better understanding of the root cause. I'm
concerned that the remaining active plex on the new array may become
inconsistent during the plex disassociation, which could cause serious
problems for our database.

Here's a little more information:

racnyc05:~>modinfo | grep -i vx
 39 7bea2000  3e4e0 307   1  vxdmp (VxVM 5.0MP3: DMP Driver)
 41 7ba00000 209248 272   1  vxio (VxVM 5.0MP3 I/O driver)
 43 7bea11e8    c78 273   1  vxspec (VxVM 5.0MP3 control/status driv)
236 7afc54a8    cb0 310   1  vxportal (VxFS 5.0_REV-5.0MP3A25_sol port)
240 7a600000 1d89e0  20   1  vxfs (VxFS 5.0_REV-5.0MP3A25_sol SunO)
260 7a7ec000   a9e0 311   1  fdd (VxQIO 5.0_REV-5.0MP3A25_sol Qui)
264 7ab4a000  51c10 315   1  vxfen (VRTS Fence 5.0MP3)
265 7b600000  21ec0 316   1  vxglm (VxGLM 5.0MP3 (SunOS 5.10))
266 7ab9e000   5418 317   1  vxgms (VxGMS 5.0MP3 (SunOS))


Let me know if you need any additional information.

Cheers,

Paul

_______________________________________________
Veritas-vx maillist  -  Veritas-vx <at> mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-vx




--
William (Bill) Havey
Symantec Certified HA Professional
917 515-3385
 

<div>
<div>Paul,</div>
<div>&nbsp;</div>
<div>Before you disassociate the plex, did you run a vxtask list|monitor to determine that the mirroring process has actually completed?</div>
<div>&nbsp;</div>
<div>Bill<br><br>
</div>
<div class="gmail_quote">
On Wed, Jul 13, 2011 at 12:48 PM, Paul Robertson <span dir="ltr">&lt;<a href="mailto:paul.marc.robertson <at> gmail.com">paul.marc.robertson <at> gmail.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote">
We are currently running SFORA RAC 5.0MP3 on Solaris 10 64-bit two-node cluster. We run a large 10TB<br>Oracle RAC database on this cluster, and we are in the process of VxVM mirroring the data from an<br>old EMC array to a new EMC array.<br><br>We provisioned the new EMC storage without incident, added the new disks to the existing diskgroups,<br>and the mirrored the data between arrays. As the final step in the process, we then ran "vxplex -g<br>dgname dis plex-nn", and saw the following for three separate volumes:<br><br><br>v&nbsp; arch&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ENABLED&nbsp; ACTIVE&nbsp;&nbsp; 6487707904 SELECT&nbsp; -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; fsgen<br>pl arch-01&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; arch&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ENABLED&nbsp; ACTIVE&nbsp;&nbsp; 6487707904 CONCAT&nbsp; -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; RW<br>sd 0804088C-01&nbsp; arch-01&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0804088C 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1696907008 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; c3t50000972C00C915Cd25 ENA<br>

sd 080408BC-01&nbsp; arch-01&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 080408BC 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1696907008 1696907008&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; c3t50000972C00C915Cd26 ENA<br>sd 080408EC-01&nbsp; arch-01&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 080408EC 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1696907008 <a href="tel:3393814016" target="_blank" value="+13393814016">3393814016</a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; c3t50000972C00C915Cd27 ENA<br>
sd 0804091C-01&nbsp; arch-01&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0804091C 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1396986880 5090721024&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; c3t50000972C00C915Cd28 ENA<br>
pl arch-02&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; arch&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ENABLED&nbsp; ACTIVE&nbsp;&nbsp; 6487707904 CONCAT&nbsp; -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; RW<br>sd <a href="tel:20520113-01" target="_blank" value="+12052011301">20520113-01</a>&nbsp; arch-02&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 20520113 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1428029344 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; c3t5006016844600097d8 ENA<br>
sd <a href="tel:20520061-02" target="_blank" value="+12052006102">20520061-02</a>&nbsp; arch-02&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 20520061 861984256 666538784 1428029344&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; c2t5006016244600097d12 ENA<br>
sd <a href="tel:20520121-01" target="_blank" value="+12052012101">20520121-01</a>&nbsp; arch-02&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 20520121 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1537081088 <a href="tel:2094568128" target="_blank" value="+12094568128">2094568128</a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; c2t5006016244600097d16 ENA<br>
sd <a href="tel:20520064-01" target="_blank" value="+12052006401">20520064-01</a>&nbsp; arch-02&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 20520064 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1428029344 3631649216&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; c3t5006016844600097d25 ENA<br>sd <a href="tel:20520103-01" target="_blank" value="+12052010301">20520103-01</a>&nbsp; arch-02&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 20520103 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1428029344 <a href="tel:5059678560" target="_blank" value="+15059678560">5059678560</a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; c3t5006016844600097d26 ENA<br><br>racnyc05:~&gt;sudo vxplex -g oradg dis arch-02<br>VxVM vxsync ERROR V-5-1-2321 Cannot lock into memory: Resource temporarily unavailable<br>VxVM vxplex ERROR V-5-1-10870 fsgen/vxplex: Warning: vxsync exited with exitcode 42:<br>

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Volume data may not be flushed to all plexes<br><br>Despite the warning above, it seemed to work, however, in that the<br>plex is disassociated and the filesystem is still accessible:<br><br>racnyc05:~&gt;vxprint -htg oradg arch arch-02<br>

V&nbsp; NAME&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; RVG/VSET/CO&nbsp; KSTATE&nbsp;&nbsp; STATE&nbsp;&nbsp;&nbsp; LENGTH&nbsp;&nbsp; READPOL&nbsp;&nbsp; PREFPLEX UTYPE<br>PL NAME&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; VOLUME&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; KSTATE&nbsp;&nbsp; STATE&nbsp;&nbsp;&nbsp; LENGTH&nbsp;&nbsp; LAYOUT&nbsp;&nbsp;&nbsp; NCOL/WID MODE<br>SD NAME&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; PLEX&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; DISK&nbsp;&nbsp;&nbsp;&nbsp; DISKOFFS LENGTH&nbsp;&nbsp; [COL/]OFF DEVICE&nbsp;&nbsp; MODE<br>

SV NAME&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; PLEX&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; VOLNAME&nbsp; NVOLLAYR LENGTH&nbsp;&nbsp; [COL/]OFF AM/NM&nbsp;&nbsp;&nbsp; MODE<br>SC NAME&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; PLEX&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; CACHE&nbsp;&nbsp;&nbsp; DISKOFFS LENGTH&nbsp;&nbsp; [COL/]OFF DEVICE&nbsp;&nbsp; MODE<br>DC NAME&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; PARENTVOL&nbsp;&nbsp;&nbsp; LOGVOL<br>SP NAME&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; SNAPVOL&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; DCO<br>

EX NAME&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ASSOC&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; VC&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; PERMS&nbsp;&nbsp;&nbsp; MODE&nbsp;&nbsp;&nbsp;&nbsp; STATE<br>SR NAME&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; KSTATE<br><br>pl arch-02&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; DISABLED -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 6487707904 CONCAT&nbsp; -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; RW<br>sd <a href="tel:20520113-01" target="_blank" value="+12052011301">20520113-01</a>&nbsp; arch-02&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 20520113 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1428029344 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; c3t5006016844600097d8 ENA<br>

sd <a href="tel:20520061-02" target="_blank" value="+12052006102">20520061-02</a>&nbsp; arch-02&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 20520061 861984256 666538784 1428029344 c2t5006016244600097d12 ENA<br>sd <a href="tel:20520121-01" target="_blank" value="+12052012101">20520121-01</a>&nbsp; arch-02&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 20520121 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1537081088 <a href="tel:2094568128" target="_blank" value="+12094568128">2094568128</a> c2t5006016244600097d16 ENA<br>
sd <a href="tel:20520064-01" target="_blank" value="+12052006401">20520064-01</a>&nbsp; arch-02&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 20520064 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1428029344 3631649216 c3t5006016844600097d25 ENA<br>
sd <a href="tel:20520103-01" target="_blank" value="+12052010301">20520103-01</a>&nbsp; arch-02&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 20520103 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1428029344 <a href="tel:5059678560" target="_blank" value="+15059678560">5059678560</a> c3t5006016844600097d26 ENA<br><br>v&nbsp; arch&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ENABLED&nbsp; ACTIVE&nbsp;&nbsp; 6487707904 SELECT&nbsp; -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; fsgen<br>pl arch-01&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; arch&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ENABLED&nbsp; ACTIVE&nbsp;&nbsp; 6487707904 CONCAT&nbsp; -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; RW<br>
sd 0804088C-01&nbsp; arch-01&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0804088C 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1696907008 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; c3t50000972C00C915Cd25 ENA<br>sd 080408BC-01&nbsp; arch-01&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 080408BC 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1696907008 1696907008 c3t50000972C00C915Cd26 ENA<br>sd 080408EC-01&nbsp; arch-01&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 080408EC 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1696907008 <a href="tel:3393814016" target="_blank" value="+13393814016">3393814016</a> c3t50000972C00C915Cd27 ENA<br>

sd 0804091C-01&nbsp; arch-01&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0804091C 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1396986880 5090721024 c3t50000972C00C915Cd28 ENA<br><br>racnyc05:~&gt;df -h /arch<br>Filesystem&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; size&nbsp;&nbsp; used&nbsp; avail capacity&nbsp; Mounted on<br>/dev/vx/dsk/oradg/arch<br>

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 3.0T&nbsp;&nbsp;&nbsp; 54G&nbsp;&nbsp; 2.8T&nbsp;&nbsp;&nbsp;&nbsp; 2%&nbsp;&nbsp;&nbsp; /arch<br><br>racnyc05:~&gt;sudo fstyp /dev/vx/rdsk/oradg/arch vxfs<br><br>It's happened on three volumes in two different clustered diskgroups<br>on this cluster. I'm unwilling to proceed with the more critical<br>

volumes until I get a better understanding of the root cause. I'm<br>concerned that the remaining active plex on the new array may become<br>inconsistent during the plex disassociation, which could cause serious<br>problems for our database.<br><br>Here's a little more information:<br><br>racnyc05:~&gt;modinfo | grep -i vx<br>&nbsp;39 7bea2000&nbsp; 3e4e0 307&nbsp;&nbsp; 1&nbsp; vxdmp (VxVM 5.0MP3: DMP Driver)<br>&nbsp;41 7ba00000 209248 272&nbsp;&nbsp; 1&nbsp; vxio (VxVM 5.0MP3 I/O driver)<br>&nbsp;43 7bea11e8&nbsp;&nbsp;&nbsp; c78 273&nbsp;&nbsp; 1&nbsp; vxspec (VxVM 5.0MP3 control/status driv)<br>

236 7afc54a8&nbsp;&nbsp;&nbsp; cb0 310&nbsp;&nbsp; 1&nbsp; vxportal (VxFS 5.0_REV-5.0MP3A25_sol port)<br>240 7a600000 1d89e0&nbsp; 20&nbsp;&nbsp; 1&nbsp; vxfs (VxFS 5.0_REV-5.0MP3A25_sol SunO)<br>260 7a7ec000&nbsp;&nbsp; a9e0 311&nbsp;&nbsp; 1&nbsp; fdd (VxQIO 5.0_REV-5.0MP3A25_sol Qui)<br>264 7ab4a000&nbsp; 51c10 315&nbsp;&nbsp; 1&nbsp; vxfen (VRTS Fence 5.0MP3)<br>

265 7b600000&nbsp; 21ec0 316&nbsp;&nbsp; 1&nbsp; vxglm (VxGLM 5.0MP3 (SunOS 5.10))<br>266 7ab9e000&nbsp;&nbsp; 5418 317&nbsp;&nbsp; 1&nbsp; vxgms (VxGMS 5.0MP3 (SunOS))<br><br><br>Let me know if you need any additional information.<br><br>Cheers,<br><br>
Paul<br><br>_______________________________________________<br>
Veritas-vx maillist &nbsp;- &nbsp;<a href="mailto:Veritas-vx <at> mailman.eng.auburn.edu">Veritas-vx <at> mailman.eng.auburn.edu</a><br><a href="http://mailman.eng.auburn.edu/mailman/listinfo/veritas-vx" target="_blank">http://mailman.eng.auburn.edu/mailman/listinfo/veritas-vx</a><br><br>
</blockquote>
</div>
<br><br clear="all"><br>-- <br><div>William (Bill) Havey</div>
<div>Symantec Certified HA&nbsp;Professional</div>
<div>917 515-3385</div>
<div>
<span></span><span></span>&nbsp;</div>
<br>
</div>
Permalink | Reply | 2 comments |
headers
Paul Robertson | 13 Jul 2011 22:08
Picon

Re: "vxplex dis" error "v-5-1-2321 cannot lock into memory:resource temporarily unavailable" with SFORAC 5.0MP3

On Wed, Jul 13, 2011 at 3:22 PM, William Havey <bbhavey <at> gmail.com> wrote:

Paul,
 
Before you disassociate the plex, did you run a vxtask list|monitor to determine that the mirroring process has actually completed?
 
Bill

Bill,

A good question. The plex synchronization was complete, as evidenced by the fact that both plexes in the volume were in the "ENABLED/ACTIVE" state.

In case someone else asks, the "vxplex dis" command was run on the VxVM cluster master as reported by "vxdctl -c mode". We've reproduced the error by resyncing the plexes and the disassociating again, so at least it's consistent. :-)

Any other suggestions are appreciated. We opened a case with the Symantec, and the current recommendation is to upgrade to 5.0MP3 RP5, which involves downtime we'd desperately like to avoid.

Cheers,

Paul
<div>
<p>On Wed, Jul 13, 2011 at 3:22 PM, William Havey <span dir="ltr">&lt;<a href="mailto:bbhavey <at> gmail.com" target="_blank">bbhavey <at> gmail.com</a>&gt;</span> wrote:<br></p>
<div class="gmail_quote">
<blockquote class="gmail_quote">

<div>Paul,</div>
<div>&nbsp;</div>
<div>Before you disassociate the plex, did you run a vxtask list|monitor to determine that the mirroring process has actually completed?</div>
<div>&nbsp;</div>
<div>Bill<br>
</div>
</blockquote>
<div>
<br>

Bill,<br><br>A good question. The plex synchronization was complete, as evidenced by the fact that both plexes in the volume were in the "ENABLED/ACTIVE" state.<br><br>In case someone else asks, the "vxplex dis" command was run on the VxVM cluster master as reported by "vxdctl -c mode". We've reproduced the error by resyncing the plexes and the disassociating again, so at least it's consistent. :-)<br><br>Any other suggestions are appreciated. We opened a case with the Symantec, and the current recommendation is to upgrade to 5.0MP3 RP5, which involves downtime we'd desperately like to avoid.<br><br>Cheers,<br><br>

Paul<br>
</div>
</div>
</div>
Permalink | Reply | 0 comments |
headers
Robinson, Greg | 14 Jul 2011 03:04
Picon

Re: "vxplex dis" error "v-5-1-2321 cannot lock into memory:resource temporarily unavailable" with SFORAC 5.0MP3 [SEC=UNCLASSIFIED]

UNCLASSIFIED

Hi Paul,
 
I'm not a database person, nor have I seen this error, but it sounds like oracle is driving the volume too fast for vxsync to keep up.  I would try stopping all I/O on this volume, run the unix sync command a few times, and then dissociate the plex.
 
We've done a few mirrors and dissociations, and we are a lower revision than you for vxvm, but higher for vxfs. But our volumes are not heavily used.
 
Greg.

IMPORTANT: This email remains the property of the Department of Defence and is subject to the jurisdiction of section 70 of the Crimes Act 1914. If you have received this email in error, you are requested to contact the sender and delete the email.

From: veritas-vx-bounces <at> mailman.eng.auburn.edu [mailto:veritas-vx-bounces <at> mailman.eng.auburn.edu] On Behalf Of Paul Robertson
Sent: Thursday, 14 July 2011 2:19 AM
To: veritas-vx <at> mailman.eng.auburn.edu
Subject: [Veritas-vx] "vxplex dis" error "v-5-1-2321 cannot lock into memory:resource temporarily unavailable" with SFORAC 5.0MP3

We are currently running SFORA RAC 5.0MP3 on Solaris 10 64-bit two-node cluster. We run a large 10TB
Oracle RAC database on this cluster, and we are in the process of VxVM mirroring the data from an
old EMC array to a new EMC array.

We provisioned the new EMC storage without incident, added the new disks to the existing diskgroups,
and the mirrored the data between arrays. As the final step in the process, we then ran "vxplex -g
dgname dis plex-nn", and saw the following for three separate volumes:


v  arch         -            ENABLED  ACTIVE   6487707904 SELECT  -       fsgen
pl arch-01      arch         ENABLED  ACTIVE   6487707904 CONCAT  -       RW
sd 0804088C-01  arch-01      0804088C 0        1696907008 0               c3t50000972C00C915Cd25 ENA
sd 080408BC-01  arch-01      080408BC 0        1696907008 1696907008      c3t50000972C00C915Cd26 ENA
sd 080408EC-01  arch-01      080408EC 0        1696907008 3393814016      c3t50000972C00C915Cd27 ENA
sd 0804091C-01  arch-01      0804091C 0        1396986880 5090721024      c3t50000972C00C915Cd28 ENA
pl arch-02      arch         ENABLED  ACTIVE   6487707904 CONCAT  -       RW
sd 20520113-01  arch-02      20520113 0        1428029344 0               c3t5006016844600097d8 ENA
sd 20520061-02  arch-02      20520061 861984256 666538784 1428029344      c2t5006016244600097d12 ENA
sd 20520121-01  arch-02      20520121 0        1537081088 2094568128      c2t5006016244600097d16 ENA
sd 20520064-01  arch-02      20520064 0        1428029344 3631649216      c3t5006016844600097d25 ENA
sd 20520103-01  arch-02      20520103 0        1428029344 5059678560      c3t5006016844600097d26 ENA

racnyc05:~>sudo vxplex -g oradg dis arch-02
VxVM vxsync ERROR V-5-1-2321 Cannot lock into memory: Resource temporarily unavailable
VxVM vxplex ERROR V-5-1-10870 fsgen/vxplex: Warning: vxsync exited with exitcode 42:
        Volume data may not be flushed to all plexes

Despite the warning above, it seemed to work, however, in that the
plex is disassociated and the filesystem is still accessible:

racnyc05:~>vxprint -htg oradg arch arch-02
V  NAME         RVG/VSET/CO  KSTATE   STATE    LENGTH   READPOL   PREFPLEX UTYPE
PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE
SC NAME         PLEX         CACHE    DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
DC NAME         PARENTVOL    LOGVOL
SP NAME         SNAPVOL      DCO
EX NAME         ASSOC        VC                       PERMS    MODE     STATE
SR NAME         KSTATE

pl arch-02      -            DISABLED -        6487707904 CONCAT  -        RW
sd 20520113-01  arch-02      20520113 0        1428029344 0       c3t5006016844600097d8 ENA
sd 20520061-02  arch-02      20520061 861984256 666538784 1428029344 c2t5006016244600097d12 ENA
sd 20520121-01  arch-02      20520121 0        1537081088 2094568128 c2t5006016244600097d16 ENA
sd 20520064-01  arch-02      20520064 0        1428029344 3631649216 c3t5006016844600097d25 ENA
sd 20520103-01  arch-02      20520103 0        1428029344 5059678560 c3t5006016844600097d26 ENA

v  arch         -            ENABLED  ACTIVE   6487707904 SELECT  -        fsgen
pl arch-01      arch         ENABLED  ACTIVE   6487707904 CONCAT  -        RW
sd 0804088C-01  arch-01      0804088C 0        1696907008 0       c3t50000972C00C915Cd25 ENA
sd 080408BC-01  arch-01      080408BC 0        1696907008 1696907008 c3t50000972C00C915Cd26 ENA
sd 080408EC-01  arch-01      080408EC 0        1696907008 3393814016 c3t50000972C00C915Cd27 ENA
sd 0804091C-01  arch-01      0804091C 0        1396986880 5090721024 c3t50000972C00C915Cd28 ENA

racnyc05:~>df -h /arch
Filesystem             size   used  avail capacity  Mounted on
/dev/vx/dsk/oradg/arch
                       3.0T    54G   2.8T     2%    /arch

racnyc05:~>sudo fstyp /dev/vx/rdsk/oradg/arch vxfs

It's happened on three volumes in two different clustered diskgroups
on this cluster. I'm unwilling to proceed with the more critical
volumes until I get a better understanding of the root cause. I'm
concerned that the remaining active plex on the new array may become
inconsistent during the plex disassociation, which could cause serious
problems for our database.

Here's a little more information:

racnyc05:~>modinfo | grep -i vx
 39 7bea2000  3e4e0 307   1  vxdmp (VxVM 5.0MP3: DMP Driver)
 41 7ba00000 209248 272   1  vxio (VxVM 5.0MP3 I/O driver)
 43 7bea11e8    c78 273   1  vxspec (VxVM 5.0MP3 control/status driv)
236 7afc54a8    cb0 310   1  vxportal (VxFS 5.0_REV-5.0MP3A25_sol port)
240 7a600000 1d89e0  20   1  vxfs (VxFS 5.0_REV-5.0MP3A25_sol SunO)
260 7a7ec000   a9e0 311   1  fdd (VxQIO 5.0_REV-5.0MP3A25_sol Qui)
264 7ab4a000  51c10 315   1  vxfen (VRTS Fence 5.0MP3)
265 7b600000  21ec0 316   1  vxglm (VxGLM 5.0MP3 (SunOS 5.10))
266 7ab9e000   5418 317   1  vxgms (VxGMS 5.0MP3 (SunOS))


Let me know if you need any additional information.

Cheers,

Paul
<div>
<p>UNCLASSIFIED</p>
<div dir="ltr" align="left"><span class="959204600-14072011">Hi Paul,</span></div>
<div dir="ltr" align="left">
<span class="959204600-14072011"></span>&nbsp;</div>
<div dir="ltr" align="left">
<span class="959204600-14072011">I'm not a database person, nor have I seen this error, 
but i</span><span class="959204600-14072011">t sounds like oracle is driving the volume too fast for 
vxsync to keep up.&nbsp; I would try stopping all I/O on this volume, run the 
unix sync command a few times, and then dissociate the plex.</span>
</div>
<div dir="ltr" align="left">
<span class="959204600-14072011"></span>&nbsp;</div>
<div dir="ltr" align="left"><span class="959204600-14072011">We've done a few mirrors and dissociations, and we are 
a lower revision than you for vxvm, but higher for vxfs.&nbsp;But our volumes 
are not heavily used.</span></div>
<div dir="ltr" align="left">
<span class="959204600-14072011"></span>&nbsp;</div>
<div dir="ltr" align="left"><span class="959204600-14072011">Greg.</span></div>
<br><p>IMPORTANT: This 
email remains the property of the Department of Defence and is subject to the 
jurisdiction of section 70 of the Crimes Act 1914. If you have received this 
email in error, you are requested to contact the sender and delete the email. 
<br></p>
<div class="OutlookMessageHeader" lang="en-us" dir="ltr" align="left">
From: veritas-vx-bounces <at> mailman.eng.auburn.edu 
[mailto:veritas-vx-bounces <at> mailman.eng.auburn.edu] On Behalf Of Paul 
Robertson<br>Sent: Thursday, 14 July 2011 2:19 AM<br>To: 
veritas-vx <at> mailman.eng.auburn.edu<br>Subject: [Veritas-vx] "vxplex dis" 
error "v-5-1-2321 cannot lock into memory:resource temporarily unavailable" with 
SFORAC 5.0MP3<br><br>
</div>
<div></div>We are currently running SFORA RAC 5.0MP3 on Solaris 10 64-bit 
two-node cluster. We run a large 10TB<br>Oracle RAC database on this cluster, 
and we are in the process of VxVM mirroring the data from an<br>old EMC array to 
a new EMC array.<br><br>We provisioned the new EMC storage without incident, 
added the new disks to the existing diskgroups,<br>and the mirrored the data 
between arrays. As the final step in the process, we then ran "vxplex 
-g<br>dgname dis plex-nn", and saw the following for three separate 
volumes:<br><br><br>v&nbsp; arch&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
-&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
ENABLED&nbsp; ACTIVE&nbsp;&nbsp; 6487707904 SELECT&nbsp; 
-&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; fsgen<br>pl 
arch-01&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
arch&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ENABLED&nbsp; 
ACTIVE&nbsp;&nbsp; 6487707904 CONCAT&nbsp; -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
RW<br>sd 0804088C-01&nbsp; arch-01&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0804088C 
0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1696907008 
0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
c3t50000972C00C915Cd25 ENA<br>sd 080408BC-01&nbsp; 
arch-01&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 080408BC 
0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1696907008 
1696907008&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; c3t50000972C00C915Cd26 ENA<br>sd 
080408EC-01&nbsp; arch-01&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 080408EC 
0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1696907008 
3393814016&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; c3t50000972C00C915Cd27 ENA<br>sd 
0804091C-01&nbsp; arch-01&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0804091C 
0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1396986880 
5090721024&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; c3t50000972C00C915Cd28 ENA<br>pl 
arch-02&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
arch&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ENABLED&nbsp; 
ACTIVE&nbsp;&nbsp; 6487707904 CONCAT&nbsp; -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
RW<br>sd 20520113-01&nbsp; arch-02&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 20520113 
0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1428029344 
0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
c3t5006016844600097d8 ENA<br>sd 20520061-02&nbsp; 
arch-02&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 20520061 861984256 666538784 
1428029344&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; c2t5006016244600097d12 ENA<br>sd 
20520121-01&nbsp; arch-02&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 20520121 
0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1537081088 
2094568128&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; c2t5006016244600097d16 ENA<br>sd 
20520064-01&nbsp; arch-02&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 20520064 
0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1428029344 
3631649216&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; c3t5006016844600097d25 ENA<br>sd 
20520103-01&nbsp; arch-02&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 20520103 
0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1428029344 
5059678560&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; c3t5006016844600097d26 
ENA<br><br>racnyc05:~&gt;sudo vxplex -g oradg dis arch-02<br>VxVM vxsync ERROR 
V-5-1-2321 Cannot lock into memory: Resource temporarily unavailable<br>VxVM 
vxplex ERROR V-5-1-10870 fsgen/vxplex: Warning: vxsync exited with exitcode 
42:<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Volume data may not be flushed 
to all plexes<br><br>Despite the warning above, it seemed to work, however, in 
that the<br>plex is disassociated and the filesystem is still 
accessible:<br><br>racnyc05:~&gt;vxprint -htg oradg arch arch-02<br>V&nbsp; 
NAME&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; RVG/VSET/CO&nbsp; 
KSTATE&nbsp;&nbsp; STATE&nbsp;&nbsp;&nbsp; LENGTH&nbsp;&nbsp; 
READPOL&nbsp;&nbsp; PREFPLEX UTYPE<br>PL 
NAME&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
VOLUME&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; KSTATE&nbsp;&nbsp; 
STATE&nbsp;&nbsp;&nbsp; LENGTH&nbsp;&nbsp; LAYOUT&nbsp;&nbsp;&nbsp; NCOL/WID 
MODE<br>SD NAME&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
PLEX&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
DISK&nbsp;&nbsp;&nbsp;&nbsp; DISKOFFS LENGTH&nbsp;&nbsp; [COL/]OFF 
DEVICE&nbsp;&nbsp; MODE<br>SV 
NAME&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
PLEX&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; VOLNAME&nbsp; NVOLLAYR 
LENGTH&nbsp;&nbsp; [COL/]OFF AM/NM&nbsp;&nbsp;&nbsp; MODE<br>SC 
NAME&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
PLEX&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; CACHE&nbsp;&nbsp;&nbsp; 
DISKOFFS LENGTH&nbsp;&nbsp; [COL/]OFF DEVICE&nbsp;&nbsp; MODE<br>DC 
NAME&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; PARENTVOL&nbsp;&nbsp;&nbsp; 
LOGVOL<br>SP NAME&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
SNAPVOL&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; DCO<br>EX 
NAME&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
ASSOC&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
VC&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
PERMS&nbsp;&nbsp;&nbsp; MODE&nbsp;&nbsp;&nbsp;&nbsp; STATE<br>SR 
NAME&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; KSTATE<br><br>pl 
arch-02&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
-&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; DISABLED 
-&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 6487707904 CONCAT&nbsp; 
-&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; RW<br>sd 20520113-01&nbsp; 
arch-02&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 20520113 
0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1428029344 
0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; c3t5006016844600097d8 ENA<br>sd 
20520061-02&nbsp; arch-02&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 20520061 861984256 
666538784 1428029344 c2t5006016244600097d12 ENA<br>sd 20520121-01&nbsp; 
arch-02&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 20520121 
0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1537081088 2094568128 
c2t5006016244600097d16 ENA<br>sd 20520064-01&nbsp; 
arch-02&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 20520064 
0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1428029344 3631649216 
c3t5006016844600097d25 ENA<br>sd 20520103-01&nbsp; 
arch-02&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 20520103 
0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1428029344 5059678560 
c3t5006016844600097d26 ENA<br><br>v&nbsp; 
arch&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
-&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
ENABLED&nbsp; ACTIVE&nbsp;&nbsp; 6487707904 SELECT&nbsp; 
-&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; fsgen<br>pl 
arch-01&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
arch&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ENABLED&nbsp; 
ACTIVE&nbsp;&nbsp; 6487707904 CONCAT&nbsp; 
-&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; RW<br>sd 0804088C-01&nbsp; 
arch-01&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0804088C 
0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1696907008 
0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; c3t50000972C00C915Cd25 ENA<br>sd 
080408BC-01&nbsp; arch-01&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 080408BC 
0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1696907008 1696907008 
c3t50000972C00C915Cd26 ENA<br>sd 080408EC-01&nbsp; 
arch-01&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 080408EC 
0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1696907008 3393814016 
c3t50000972C00C915Cd27 ENA<br>sd 0804091C-01&nbsp; 
arch-01&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0804091C 
0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1396986880 5090721024 
c3t50000972C00C915Cd28 ENA<br><br>racnyc05:~&gt;df -h 
/arch<br>Filesystem&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
size&nbsp;&nbsp; used&nbsp; avail capacity&nbsp; Mounted 
on<br>/dev/vx/dsk/oradg/arch<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
3.0T&nbsp;&nbsp;&nbsp; 54G&nbsp;&nbsp; 2.8T&nbsp;&nbsp;&nbsp;&nbsp; 
2%&nbsp;&nbsp;&nbsp; /arch<br><br>racnyc05:~&gt;sudo fstyp 
/dev/vx/rdsk/oradg/arch vxfs<br><br>It's happened on three volumes in two 
different clustered diskgroups<br>on this cluster. I'm unwilling to proceed with 
the more critical<br>volumes until I get a better understanding of the root 
cause. I'm<br>concerned that the remaining active plex on the new array may 
become<br>inconsistent during the plex disassociation, which could cause 
serious<br>problems for our database.<br><br>Here's a little more 
information:<br><br>racnyc05:~&gt;modinfo | grep -i vx<br>&nbsp;39 
7bea2000&nbsp; 3e4e0 307&nbsp;&nbsp; 1&nbsp; vxdmp (VxVM 5.0MP3: DMP 
Driver)<br>&nbsp;41 7ba00000 209248 272&nbsp;&nbsp; 1&nbsp; vxio (VxVM 5.0MP3 
I/O driver)<br>&nbsp;43 7bea11e8&nbsp;&nbsp;&nbsp; c78 273&nbsp;&nbsp; 1&nbsp; 
vxspec (VxVM 5.0MP3 control/status driv)<br>236 7afc54a8&nbsp;&nbsp;&nbsp; cb0 
310&nbsp;&nbsp; 1&nbsp; vxportal (VxFS 5.0_REV-5.0MP3A25_sol port)<br>240 
7a600000 1d89e0&nbsp; 20&nbsp;&nbsp; 1&nbsp; vxfs (VxFS 5.0_REV-5.0MP3A25_sol 
SunO)<br>260 7a7ec000&nbsp;&nbsp; a9e0 311&nbsp;&nbsp; 1&nbsp; fdd (VxQIO 
5.0_REV-5.0MP3A25_sol Qui)<br>264 7ab4a000&nbsp; 51c10 315&nbsp;&nbsp; 1&nbsp; 
vxfen (VRTS Fence 5.0MP3)<br>265 7b600000&nbsp; 21ec0 316&nbsp;&nbsp; 1&nbsp; 
vxglm (VxGLM 5.0MP3 (SunOS 5.10))<br>266 7ab9e000&nbsp;&nbsp; 5418 
317&nbsp;&nbsp; 1&nbsp; vxgms (VxGMS 5.0MP3 (SunOS))<br><br><br>Let me know if 
you need any additional 
information.<br><br>Cheers,<br><br>Paul<br>
</div>
Permalink | Reply | 0 comments |
headers
Stuart Andrews | 14 Jul 2011 04:24
Favicon

Re: "vxplex dis" error "v-5-1-2321 cannot lock into memory:resource temporarily unavailable" with SFORAC 5.0MP3

Paul

Two things –  from errno.h

#define EUNATCH 42      /* Protocol driver not attached         */

But I have no idea what it specifically means – may be related to the kernel modules and VxVM internals.

 

Secondly – why not make a vol from the dissociated plex

# vxmake –g oradg –U fsgen vol TEST_arch plex=arch-02

Repeat for other vols with name changes as required.  Then set up your database init.ora files to point to these new objects.

Then fsck and mount and fire up the database in read only mode to check consistency etc.

Stuart

 

From: veritas-vx-bounces <at> mailman.eng.auburn.edu [mailto:veritas-vx-bounces <at> mailman.eng.auburn.edu] On Behalf Of William Havey
Sent: Thursday, 14 July 2011 5:22 AM
To: Paul Robertson
Cc: veritas-vx <at> mailman.eng.auburn.edu
Subject: Re: [Veritas-vx] "vxplex dis" error "v-5-1-2321 cannot lock into memory:resource temporarily unavailable" with SFORAC 5.0MP3

 

Paul,

 

Before you disassociate the plex, did you run a vxtask list|monitor to determine that the mirroring process has actually completed?

 

Bill

On Wed, Jul 13, 2011 at 12:48 PM, Paul Robertson <paul.marc.robertson <at> gmail.com> wrote:

We are currently running SFORA RAC 5.0MP3 on Solaris 10 64-bit two-node cluster. We run a large 10TB
Oracle RAC database on this cluster, and we are in the process of VxVM mirroring the data from an
old EMC array to a new EMC array.

We provisioned the new EMC storage without incident, added the new disks to the existing diskgroups,
and the mirrored the data between arrays. As the final step in the process, we then ran "vxplex -g
dgname dis plex-nn", and saw the following for three separate volumes:


v  arch         -            ENABLED  ACTIVE   6487707904 SELECT  -       fsgen
pl arch-01      arch         ENABLED  ACTIVE   6487707904 CONCAT  -       RW
sd 0804088C-01  arch-01      0804088C 0        1696907008 0               c3t50000972C00C915Cd25 ENA
sd 080408BC-01  arch-01      080408BC 0        1696907008 1696907008      c3t50000972C00C915Cd26 ENA
sd 080408EC-01  arch-01      080408EC 0        1696907008 3393814016      c3t50000972C00C915Cd27 ENA
sd 0804091C-01  arch-01      0804091C 0        1396986880 5090721024      c3t50000972C00C915Cd28 ENA
pl arch-02      arch         ENABLED  ACTIVE   6487707904 CONCAT  -       RW
sd 20520113-01  arch-02      20520113 0        1428029344 0               c3t5006016844600097d8 ENA
sd 20520061-02  arch-02      20520061 861984256 666538784 1428029344      c2t5006016244600097d12 ENA
sd 20520121-01  arch-02      20520121 0        1537081088 2094568128      c2t5006016244600097d16 ENA
sd 20520064-01  arch-02      20520064 0        1428029344 3631649216      c3t5006016844600097d25 ENA
sd 20520103-01  arch-02      20520103 0        1428029344 5059678560      c3t5006016844600097d26 ENA

racnyc05:~>sudo vxplex -g oradg dis arch-02
VxVM vxsync ERROR V-5-1-2321 Cannot lock into memory: Resource temporarily unavailable
VxVM vxplex ERROR V-5-1-10870 fsgen/vxplex: Warning: vxsync exited with exitcode 42:
        Volume data may not be flushed to all plexes

Despite the warning above, it seemed to work, however, in that the
plex is disassociated and the filesystem is still accessible:

racnyc05:~>vxprint -htg oradg arch arch-02
V  NAME         RVG/VSET/CO  KSTATE   STATE    LENGTH   READPOL   PREFPLEX UTYPE
PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE
SC NAME         PLEX         CACHE    DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
DC NAME         PARENTVOL    LOGVOL
SP NAME         SNAPVOL      DCO
EX NAME         ASSOC        VC                       PERMS    MODE     STATE
SR NAME         KSTATE

pl arch-02      -            DISABLED -        6487707904 CONCAT  -        RW
sd 20520113-01  arch-02      20520113 0        1428029344 0       c3t5006016844600097d8 ENA
sd 20520061-02  arch-02      20520061 861984256 666538784 1428029344 c2t5006016244600097d12 ENA
sd 20520121-01  arch-02      20520121 0        1537081088 2094568128 c2t5006016244600097d16 ENA
sd 20520064-01  arch-02      20520064 0        1428029344 3631649216 c3t5006016844600097d25 ENA
sd 20520103-01  arch-02      20520103 0        1428029344 5059678560 c3t5006016844600097d26 ENA

v  arch         -            ENABLED  ACTIVE   6487707904 SELECT  -        fsgen
pl arch-01      arch         ENABLED  ACTIVE   6487707904 CONCAT  -        RW
sd 0804088C-01  arch-01      0804088C 0        1696907008 0       c3t50000972C00C915Cd25 ENA
sd 080408BC-01  arch-01      080408BC 0        1696907008 1696907008 c3t50000972C00C915Cd26 ENA
sd 080408EC-01  arch-01      080408EC 0        1696907008 3393814016 c3t50000972C00C915Cd27 ENA
sd 0804091C-01  arch-01      0804091C 0        1396986880 5090721024 c3t50000972C00C915Cd28 ENA

racnyc05:~>df -h /arch
Filesystem             size   used  avail capacity  Mounted on
/dev/vx/dsk/oradg/arch
                       3.0T    54G   2.8T     2%    /arch

racnyc05:~>sudo fstyp /dev/vx/rdsk/oradg/arch vxfs

It's happened on three volumes in two different clustered diskgroups
on this cluster. I'm unwilling to proceed with the more critical
volumes until I get a better understanding of the root cause. I'm
concerned that the remaining active plex on the new array may become
inconsistent during the plex disassociation, which could cause serious
problems for our database.

Here's a little more information:

racnyc05:~>modinfo | grep -i vx
 39 7bea2000  3e4e0 307   1  vxdmp (VxVM 5.0MP3: DMP Driver)
 41 7ba00000 209248 272   1  vxio (VxVM 5.0MP3 I/O driver)
 43 7bea11e8    c78 273   1  vxspec (VxVM 5.0MP3 control/status driv)
236 7afc54a8    cb0 310   1  vxportal (VxFS 5.0_REV-5.0MP3A25_sol port)
240 7a600000 1d89e0  20   1  vxfs (VxFS 5.0_REV-5.0MP3A25_sol SunO)
260 7a7ec000   a9e0 311   1  fdd (VxQIO 5.0_REV-5.0MP3A25_sol Qui)
264 7ab4a000  51c10 315   1  vxfen (VRTS Fence 5.0MP3)
265 7b600000  21ec0 316   1  vxglm (VxGLM 5.0MP3 (SunOS 5.10))
266 7ab9e000   5418 317   1  vxgms (VxGMS 5.0MP3 (SunOS))


Let me know if you need any additional information.

Cheers,

Paul

_______________________________________________
Veritas-vx maillist  -  Veritas-vx <at> mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-vx




--

William (Bill) Havey

Symantec Certified HA Professional

917 515-3385

 

 

<div><div class="WordSection1">
<p class="MsoNormal"><span>Paul<p></p></span></p>
<p class="MsoNormal"><span>Two things &ndash; &nbsp;from errno.h<p></p></span></p>
<p class="MsoNormal"><span>#define EUNATCH 42&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;/* Protocol driver not attached&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; */<p></p></span></p>
<p class="MsoNormal"><span>But I have no idea what it specifically means &ndash; may be related to the kernel modules and VxVM internals.<p></p></span></p>
<p class="MsoNormal"><span><p>&nbsp;</p></span></p>
<p class="MsoNormal"><span>Secondly &ndash; why not make a vol from the dissociated plex <p></p></span></p>
<p class="MsoNormal"><span># vxmake &ndash;g oradg &ndash;U fsgen vol TEST_arch plex=arch-02<p></p></span></p>
<p class="MsoNormal"><span>Repeat for other vols with name changes as required.&nbsp; Then set up your database init.ora files to point to these new objects.<p></p></span></p>
<p class="MsoNormal"><span>Then fsck and mount and fire up the database in read only mode to check consistency etc.<p></p></span></p>
<p class="MsoNormal"><span>Stuart</span><span lang="EN-AU"><p></p></span></p>
<p class="MsoNormal"><span><p>&nbsp;</p></span></p>
<div><p class="MsoNormal"><span>From:</span><span> veritas-vx-bounces <at> mailman.eng.auburn.edu [mailto:veritas-vx-bounces <at> mailman.eng.auburn.edu] On Behalf Of William Havey<br>Sent: Thursday, 14 July 2011 5:22 AM<br>To: Paul Robertson<br>Cc: veritas-vx <at> mailman.eng.auburn.edu<br>Subject: Re: [Veritas-vx] "vxplex dis" error "v-5-1-2321 cannot lock into memory:resource temporarily unavailable" with SFORAC 5.0MP3<p></p></span></p></div>
<p class="MsoNormal"><p>&nbsp;</p></p>
<div><p class="MsoNormal">Paul,<p></p></p></div>
<div><p class="MsoNormal">&nbsp;<p></p></p></div>
<div><p class="MsoNormal">Before you disassociate the plex, did you run a vxtask list|monitor to determine that the mirroring process has actually completed?<p></p></p></div>
<div><p class="MsoNormal">&nbsp;<p></p></p></div>
<div><p class="MsoNormal">Bill<p></p></p></div>
<div>
<p class="MsoNormal">On Wed, Jul 13, 2011 at 12:48 PM, Paul Robertson &lt;<a href="mailto:paul.marc.robertson <at> gmail.com">paul.marc.robertson <at> gmail.com</a>&gt; wrote:<p></p></p>
<p class="MsoNormal">We are currently running SFORA RAC 5.0MP3 on Solaris 10 64-bit two-node cluster. We run a large 10TB<br>Oracle RAC database on this cluster, and we are in the process of VxVM mirroring the data from an<br>old EMC array to a new EMC array.<br><br>We provisioned the new EMC storage without incident, added the new disks to the existing diskgroups,<br>and the mirrored the data between arrays. As the final step in the process, we then ran "vxplex -g<br>dgname dis plex-nn", and saw the following for three separate volumes:<br><br><br>v&nbsp; arch&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ENABLED&nbsp; ACTIVE&nbsp;&nbsp; 6487707904 SELECT&nbsp; -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; fsgen<br>pl arch-01&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; arch&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ENABLED&nbsp; ACTIVE&nbsp;&nbsp; 6487707904 CONCAT&nbsp; -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; RW<br>sd 0804088C-01&nbsp; arch-01&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0804088C 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1696907008 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; c3t50000972C00C915Cd25 ENA<br>sd 080408BC-01&nbsp; arch-01&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 080408BC 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1696907008 1696907008&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; c3t50000972C00C915Cd26 ENA<br>sd 080408EC-01&nbsp; arch-01&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 080408EC 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1696907008 <a href="tel:3393814016" target="_blank">3393814016</a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; c3t50000972C00C915Cd27 ENA<br>sd 0804091C-01&nbsp; arch-01&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0804091C 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1396986880 5090721024&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; c3t50000972C00C915Cd28 ENA<br>pl arch-02&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; arch&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ENABLED&nbsp; ACTIVE&nbsp;&nbsp; 6487707904 CONCAT&nbsp; -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; RW<br>sd <a href="tel:20520113-01" target="_blank">20520113-01</a>&nbsp; arch-02&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 20520113 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1428029344 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; c3t5006016844600097d8 ENA<br>sd <a href="tel:20520061-02" target="_blank">20520061-02</a>&nbsp; arch-02&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 20520061 861984256 666538784 1428029344&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; c2t5006016244600097d12 ENA<br>sd <a href="tel:20520121-01" target="_blank">20520121-01</a>&nbsp; arch-02&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 20520121 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1537081088 <a href="tel:2094568128" target="_blank">2094568128</a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; c2t5006016244600097d16 ENA<br>sd <a href="tel:20520064-01" target="_blank">20520064-01</a>&nbsp; arch-02&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 20520064 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1428029344 3631649216&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; c3t5006016844600097d25 ENA<br>sd <a href="tel:20520103-01" target="_blank">20520103-01</a>&nbsp; arch-02&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 20520103 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1428029344 <a href="tel:5059678560" target="_blank">5059678560</a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; c3t5006016844600097d26 ENA<br><br>racnyc05:~&gt;sudo vxplex -g oradg dis arch-02<br>VxVM vxsync ERROR V-5-1-2321 Cannot lock into memory: Resource temporarily unavailable<br>VxVM vxplex ERROR V-5-1-10870 fsgen/vxplex: Warning: vxsync exited with exitcode 42:<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Volume data may not be flushed to all plexes<br><br>Despite the warning above, it seemed to work, however, in that the<br>plex is disassociated and the filesystem is still accessible:<br><br>racnyc05:~&gt;vxprint -htg oradg arch arch-02<br>V&nbsp; NAME&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; RVG/VSET/CO&nbsp; KSTATE&nbsp;&nbsp; STATE&nbsp;&nbsp;&nbsp; LENGTH&nbsp;&nbsp; READPOL&nbsp;&nbsp; PREFPLEX UTYPE<br>PL NAME&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; VOLUME&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; KSTATE&nbsp;&nbsp; STATE&nbsp;&nbsp;&nbsp; LENGTH&nbsp;&nbsp; LAYOUT&nbsp;&nbsp;&nbsp; NCOL/WID MODE<br>SD NAME&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; PLEX&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; DISK&nbsp;&nbsp;&nbsp;&nbsp; DISKOFFS LENGTH&nbsp;&nbsp; [COL/]OFF DEVICE&nbsp;&nbsp; MODE<br>SV NAME&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; PLEX&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; VOLNAME&nbsp; NVOLLAYR LENGTH&nbsp;&nbsp; [COL/]OFF AM/NM&nbsp;&nbsp;&nbsp; MODE<br>SC NAME&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; PLEX&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; CACHE&nbsp;&nbsp;&nbsp; DISKOFFS LENGTH&nbsp;&nbsp; [COL/]OFF DEVICE&nbsp;&nbsp; MODE<br>DC NAME&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; PARENTVOL&nbsp;&nbsp;&nbsp; LOGVOL<br>SP NAME&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; SNAPVOL&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; DCO<br>EX NAME&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ASSOC&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; VC&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; PERMS&nbsp;&nbsp;&nbsp; MODE&nbsp;&nbsp;&nbsp;&nbsp; STATE<br>SR NAME&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; KSTATE<br><br>pl arch-02&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; DISABLED -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 6487707904 CONCAT&nbsp; -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; RW<br>sd <a href="tel:20520113-01" target="_blank">20520113-01</a>&nbsp; arch-02&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 20520113 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1428029344 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; c3t5006016844600097d8 ENA<br>sd <a href="tel:20520061-02" target="_blank">20520061-02</a>&nbsp; arch-02&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 20520061 861984256 666538784 1428029344 c2t5006016244600097d12 ENA<br>sd <a href="tel:20520121-01" target="_blank">20520121-01</a>&nbsp; arch-02&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 20520121 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1537081088 <a href="tel:2094568128" target="_blank">2094568128</a> c2t5006016244600097d16 ENA<br>sd <a href="tel:20520064-01" target="_blank">20520064-01</a>&nbsp; arch-02&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 20520064 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1428029344 3631649216 c3t5006016844600097d25 ENA<br>sd <a href="tel:20520103-01" target="_blank">20520103-01</a>&nbsp; arch-02&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 20520103 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1428029344 <a href="tel:5059678560" target="_blank">5059678560</a> c3t5006016844600097d26 ENA<br><br>v&nbsp; arch&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ENABLED&nbsp; ACTIVE&nbsp;&nbsp; 6487707904 SELECT&nbsp; -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; fsgen<br>pl arch-01&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; arch&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ENABLED&nbsp; ACTIVE&nbsp;&nbsp; 6487707904 CONCAT&nbsp; -&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; RW<br>sd 0804088C-01&nbsp; arch-01&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0804088C 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1696907008 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; c3t50000972C00C915Cd25 ENA<br>sd 080408BC-01&nbsp; arch-01&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 080408BC 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1696907008 1696907008 c3t50000972C00C915Cd26 ENA<br>sd 080408EC-01&nbsp; arch-01&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 080408EC 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1696907008 <a href="tel:3393814016" target="_blank">3393814016</a> c3t50000972C00C915Cd27 ENA<br>sd 0804091C-01&nbsp; arch-01&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0804091C 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1396986880 5090721024 c3t50000972C00C915Cd28 ENA<br><br>racnyc05:~&gt;df -h /arch<br>Filesystem&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; size&nbsp;&nbsp; used&nbsp; avail capacity&nbsp; Mounted on<br>/dev/vx/dsk/oradg/arch<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 3.0T&nbsp;&nbsp;&nbsp; 54G&nbsp;&nbsp; 2.8T&nbsp;&nbsp;&nbsp;&nbsp; 2%&nbsp;&nbsp;&nbsp; /arch<br><br>racnyc05:~&gt;sudo fstyp /dev/vx/rdsk/oradg/arch vxfs<br><br>It's happened on three volumes in two different clustered diskgroups<br>on this cluster. I'm unwilling to proceed with the more critical<br>volumes until I get a better understanding of the root cause. I'm<br>concerned that the remaining active plex on the new array may become<br>inconsistent during the plex disassociation, which could cause serious<br>problems for our database.<br><br>Here's a little more information:<br><br>racnyc05:~&gt;modinfo | grep -i vx<br>&nbsp;39 7bea2000&nbsp; 3e4e0 307&nbsp;&nbsp; 1&nbsp; vxdmp (VxVM 5.0MP3: DMP Driver)<br>&nbsp;41 7ba00000 209248 272&nbsp;&nbsp; 1&nbsp; vxio (VxVM 5.0MP3 I/O driver)<br>&nbsp;43 7bea11e8&nbsp;&nbsp;&nbsp; c78 273&nbsp;&nbsp; 1&nbsp; vxspec (VxVM 5.0MP3 control/status driv)<br>236 7afc54a8&nbsp;&nbsp;&nbsp; cb0 310&nbsp;&nbsp; 1&nbsp; vxportal (VxFS 5.0_REV-5.0MP3A25_sol port)<br>240 7a600000 1d89e0&nbsp; 20&nbsp;&nbsp; 1&nbsp; vxfs (VxFS 5.0_REV-5.0MP3A25_sol SunO)<br>260 7a7ec000&nbsp;&nbsp; a9e0 311&nbsp;&nbsp; 1&nbsp; fdd (VxQIO 5.0_REV-5.0MP3A25_sol Qui)<br>264 7ab4a000&nbsp; 51c10 315&nbsp;&nbsp; 1&nbsp; vxfen (VRTS Fence 5.0MP3)<br>265 7b600000&nbsp; 21ec0 316&nbsp;&nbsp; 1&nbsp; vxglm (VxGLM 5.0MP3 (SunOS 5.10))<br>266 7ab9e000&nbsp;&nbsp; 5418 317&nbsp;&nbsp; 1&nbsp; vxgms (VxGMS 5.0MP3 (SunOS))<br><br><br>Let me know if you need any additional information.<br><br>Cheers,<br><span><br>Paul<br></span><br>_______________________________________________<br>Veritas-vx maillist &nbsp;- &nbsp;<a href="mailto:Veritas-vx <at> mailman.eng.auburn.edu">Veritas-vx <at> mailman.eng.auburn.edu</a><br><a href="http://mailman.eng.auburn.edu/mailman/listinfo/veritas-vx" target="_blank">http://mailman.eng.auburn.edu/mailman/listinfo/veritas-vx</a><p></p></p>
</div>
<p class="MsoNormal"><br><br clear="all"><br>-- <p></p></p>
<div><p class="MsoNormal">William (Bill) Havey<p></p></p></div>
<div><p class="MsoNormal">Symantec Certified HA&nbsp;Professional<p></p></p></div>
<div><p class="MsoNormal">917 515-3385<p></p></p></div>
<div><p class="MsoNormal">&nbsp;<p></p></p></div>
<p class="MsoNormal"><p>&nbsp;</p></p>
</div></div>
Permalink | Reply | 0 comments |
headers
Carl E. Ma | 31 Jul 2011 18:20
Picon
Favicon

vxfs minimum reserve space

Hi,

From the
archive(http://www.mail-archive.com/veritas-vx <at> mailman.eng.auburn.edu/msg01550.html), vxfs
should have the internal reserved space to continue filesystem operation even it is 100% full and the
exception should be very rare.

We are running vxfs 5.1sp1 on solaris 10_x86. After filesystem becomes 100% full, we can't rm/mv/unlink
files to free space. The only solution is umount filesystem/fsck/remount it, which introduced
filesystem outage. I don't see this behavior on ufs/zfs or even ext3.

Maybe I miss some configuration parameters? Can anyone have suggestions?

Thanks,

zhu
Permalink | Reply | 0 comments |

Gmane