Internal Production:
Last Updated 2006-08-09
Managing Mirrors with DiskSuite 4.2.1
introduction to DiskSuite and mirrors
=====================================
For systems with one or more simple standalone disks, Solstice
DiskSuite (bundled with Solaris) provides software implementations of
RAID levels 0 (concatenations and stripes), 1 (mirrors), and 5 (striping
with parity). File systems may be grown dynamically; logical disk
partitions may span arbitrary physical disk partitions on separate
disks; unused partitions may be automatically hot-swapped to replace
failed partitions. In general, data and file systems may be variously
mapped to disk(s) to provide increased capacity, availability, and I/O
performance.
DiskSuite accomplishes these tasks through the creation of various
types of metadevices. To applications, these metadevices are accessed
just as physical devices are accessed, with DiskSuite orchestrating I/O
to the underlying physical devices. Perhaps the best way to understand
how this works is to look at an example, that of mirroring, the main
subject of this document.
Consider a system, representative of many of ARSC's Suns, consisting
of two disks, and let's look at how DiskSuite would set up a mirror for
the root partition of the primary (boot) disk of this system. For this
system, the device paths for the two root disk partitions are:
/dev/dsk/c0t0d0s0 # Root partition on primary (boot) disk.
/dev/dsk/c0t1d0s0 # Corresponding partition on secondary disk.
The goal here is to mirror on the secondary disk all writes made to the
primary disk, so that in the event of failure of one of these disks, the
other disk can carry on in a transparent manner with no loss of data or
service. What we do here is have DiskSuite create a mirror metadevice
to replace references to these two physical devices. The path to this
new (logical) metadevice would look something like this:
/dev/md/dsk/d10
Applications accessing this device neither know nor need to know which
underlying physical device it is reading from or writing to. By default,
writes will occur in parallel to both devices, while reads will
alternate (round-robin) between them.
Metadevices are also created for the two underlying physical devices,
which are now referred to as submirrors. DiskSuite allows for up to
three submirrors, so that one submirror can be taken offline - say for
making a backup of, or cloning - while the remaining two submirrors
would continue to supply mirroring. In this discussion, however, we will
discuss just two-way mirroring.
These two submirrors would be given names such as:
/dev/md/dsk/d20
/dev/md/dsk/d30
So now we have a logical two-way mirror, d10, which consists of two
submirrors, d20 and d30, each of which corresponds to actual physical
disk partitions. (Note that this need not be so; the submirrors could,
for example, be concatentations of partitions on different disks, if
desired.)
Initially, before these disks become mirrors of each other, a decision
must be made as to which disk is the 'master', to which the other disk
will sync. Note that to 'sync to' here means to 'make into copy of', so
that sync'ing diskX to diskY results in diskX becoming a copy of diskY.
This distinction loses importance once mirroring is implemented, but is
essential in setting up the mirror, as in most cases the two disks to be
mirrored will initially be out of sync.
This simple configuration, of one mirror metadevice, and two submirror
metadevices, is stored in a metadatabase, one of DiskSuite's most
important objects which, among other things, stores the mapping of
logical to physical devices. So important are these metadatabases, that
DiskSuite requires that multiple copies be created, in disk partitions
dedicated for their exclusive use, ideally distributed across multiple
disks and controllers for fault tolerance.
When creating or revising metadevice configurations, DiskSuite will
update its metadatabases. However, these metadatabases must be created
before using DiskSuite for the first time.
Following are details of an example mirror, showing first how the
metadatabases are created (empty), then how the example mirror is
created. Following that is an example for breaking the mirror. Following
that are some other operations that may be performed on mirrors, and a
general troubleshooting section.
general notes
=============
- What follows works for version 4.2.1 of DiskSuite on Solaris 8, but
may not work for other DiskSuite versions, such as the earlier 4.2
release (which came with Solaris 7 and 2.6).
- Page references in the following notes are to:
_Solstice DiskSuite 4.2.1 User's Guide_ (Feb 2000)
a copy of which resides in the Tech Services library, and which (along
with the companion "Reference Guide") served as the source for most of
this document.
- DiskSuite commands involving the making and breaking of mirrors must
be run as superuser; however, do not use sudo, as sudo depends on (at
least) the /var and /export file systems being mounted, and will fail
if they are not. In these procedures, safety lies in becoming root for
the duration of the procedure.
- After making any change to metadevice configuration (creating/deleting
/revising), be sure to backcopy the newly created /etc/lvm/md.cf file
to SunFiles on the admin machine. This file will enable you to
recreate lost metadatabases. In the absence of valid metadatabases,
all data on the metadevices configured in these metadatabases may be
totally lost.
- /etc/vfstab governs file system defaults, and each of the procedures
involving the making or breaking of mirrors requires that this file be
changed so that actual physical slices are replaced with new
metadevices, or vice versa. For example, the following line from
/etc/vfstab:
/dev/dsk/c0t0d0s6 /dev/rdsk/c0t0d0s6 ...
may be changed to:
/dev/md/dsk/d16 /dev/md/rdsk/d16 ...
or vice versa. If making or breaking the root mirror, the metaroot
tool will make this change for you (and will also update /etc/system);
otherwise, use your favorite editor. Note that vi expects to find
/var/tmp; if /var is unmounted, simply mkdir /var/tmp. If you corrupt
vfstab, or fail to run metaroot when required, see 'troubleshooting'
below.
- The procedures for making/breaking mirrors call for unmounting those
file systems which can be unmounted on a running system. If the umount
command fails, use lsof to see which processes are using the target
file system, kill them as appropriate, then reissue umount with '-f'.
- If you encounter a "stale database" error message while issuing any
DiskSuite command, see the troubleshooting section below.
mirror configuration example (from Sol. 8 baseline disk)
========================================================
d10 - mirror of /, composed of two submirrors:
d20 - c0t0d0s0 (master)
d30 - c0t1d0s0 (replica)
d11 - mirror of /usr, composed of two submirrors:
d21 - c0t0d0s1 (master)
d31 - c0t1d0s1 (replica)
d14 - mirror of /var, composed of two submirrors:
d24 - c0t0d0s4 (master)
d34 - c0t1d0s4 (replica)
d15 - mirror of /export, composed of two submirrors:
d25 - c0t0d0s5 (master)
d35 - c0t1d0s5 (replica)
d16 - mirror of swap, composed of two submirrors:
d26 - c0t0d0s6 (master)
d36 - c0t1d0s6 (replica)
relevant lines from corresponding /etc/vfstab (using metadevices)
-----------------------------------------------------------------
/dev/md/dsk/d10 /dev/md/rdsk/d10 / ufs 1 no -
/dev/md/dsk/d11 /dev/md/rdsk/d11 /usr ufs 1 no logging
/dev/md/dsk/d14 /dev/md/rdsk/d14 /var ufs 1 no logging
/dev/md/dsk/d15 /dev/md/rdsk/d15 /export ufs 1 yes logging
/dev/md/dsk/d16 - - swap - no -
relevant lines from previous /etc/vfstab (using raw devices)
------------------------------------------------------------
/dev/dsk/c0t0d0s0 /dev/rdsk/c0t0d0s0 / ufs 1 no -
/dev/dsk/c0t0d0s1 /dev/rdsk/c0t0d0s1 /usr ufs 1 no logging
/dev/dsk/c0t0d0s4 /dev/rdsk/c0t0d0s4 /var ufs 1 no logging
/dev/dsk/c0t0d0s5 /dev/rdsk/c0t0d0s5 /export ufs 1 yes logging
/dev/dsk/c0t0d0s6 - - swap - no -
Notes for example mirror configuration
--------------------------------------
- Slices for which metadevices are not created:
- 2: 'backup' (whole disk slice)
- 3: 'unassigned' (unused)
- 7: 'unassigned' (but used by DiskSuite for its metadevice databases)
- Numbering scheme (arbitrary, but constrained to choices of those
listed in /dev/md/dsk):
- 'd' followed by 2 digits, where:
- second digit always indicates disk slice number
- first digit (which cannot be 0) indicates:
1 for mirror device (of corresponding slice) for diskA
2 for master submirror (of corresponding slice) for diskA
3 for replica submirror (of corresponding slice) for diskA
4 for mirror device (of corresponding slice) for diskB (if any)
5 for master submirror (of corresponding slice) for diskB (if any)
6 for replica submirror (of corresponding slice) for diskB (if any)
7 for mirror device (of corresponding slice) for diskC (if any)
8 for master submirror (of corresponding slice) for diskC (if any)
9 for replica submirror (of corresponding slice) for diskC (if any)
metadatabases (p. 43)
=====================
DiskSuite stores metadevice configuration and state information in
metadatabases. These metadatabases are required for DiskSuite to
function, and must be created before using DiskSuite for the first time.
A minimum of three metadabases must be created initially, up to a
maximum of fifty. In normal operation, all metadatabases are kept in
sync. Having multiple metadatabases provides redundancy, especially
when these metadatabases are distributed across multiple disks and
controllers.
Recommended configurations:
---------------------------
number of disks recommended number of metadatabases
--------------- -----------------------------------
1 3
2 - 4 2 on each disk
5 or more 1 on each disk
Degraded configurations (corrupted or unavailable metadatabases):
-----------------------------------------------------------------
no databases: catastrophic loss of data in corresponding metadevices
less than half: system panic
* exactly half: running system will continue running, but will not
reboot normally (will halt at single-user for repair)
more than half: runs and reboots normally; degraded databases flagged
* Addition of the following line to /etc/system:
set md:mirrored_root_flag=1
will enable a system with just one half of its extant metadatabases
functional to boot without halting, and is recommended when a boot
disk is a part of the metadevice governed by these metadatabases.
metadatabase partitions
-----------------------
A metadatabase occupies 1034 disk blocks (517KB), beginning at block
16. The following example shows four metadatabases written consecutively
to disk partition 7, beginning at block 16:
# metadb
flags first blk block count
a m p luo 16 1034 /dev/dsk/c0t0d0s7
a p luo 1050 1034 /dev/dsk/c0t0d0s7
a p luo 2084 1034 /dev/dsk/c0t0d0s7
a p luo 3118 1034 /dev/dsk/c0t0d0s7
These metadatabases consist of raw data and will overwrite whatever may
already occupy these blocks. For this reason, they must not be created
on any partition already storing data or metadata. Writing metadata to a
partition on which a file system has been constructed (even if empty),
for example, will overwrite the initial file system blocks, resulting in
serious file system corruption and likely loss of data.
Metadatabases may, however, be written to a currently unused partition
which may then later be incorporated into a metadevice storing file
system data (except for the /, /usr or swap file systems). In this case,
DiskSuite recognizes the preexistence of the metadatabases and offsets
its I/O of file data accordingly. This economical use of disk partitions
has the disadvantage, however, in that a partition created in this
manner can then only be accessed through DiskSuite.
Instead, metadatabases are normally created on unused disk partitions
dedicated to them. If there are unused partitions and unused disk space
to allot to one of these unused partitions, then the procedure is
straightforward. If there are unused partitions but no unassigned disk
space, and if there is a swap partition, then space can be (dynamically)
reclaimed from the swap partition and given to an unused partition:
init 1 # Go to single-user.
swap -l # Identify swap slice.
swap -d /dev/dsk/<swapslice> # Disable swapping to that slice.
format # Make new slice with space from swap.
Note that since only at most three metadatabases need to be created for
ample redundancy, a partition need be at most only a few megabytes, and
that since the space requirement is minimal, the only real problem may
lie in obtaining an unused disk partition.
creating metadatabases
----------------------
Metadatabases can only be created on a disk slice not already
containing metadatabases. So, first issue:
metadb
and see if there are any preexisting metadatabases. If there are, and if
you are certain you wish to replace them, you must first delete the
existing metadatabases; see the next section.
If you are creating metadatabases on a new disk slice not already
containing metadatabases, then simply issue (for example):
metadb -a -f -c 3 c0t0d0s7
This command creates three metadatabases on disk partition 7. The 'c'
flag specifies the count, while the 'a' and 'f' flags specify the
forcible attachment of these new metadatabases, required for creation of
initial metadatabases. This example should only be used when DiskSuite
will be managing a single disk, as it creates a single point of failure
(all the metadatabases on one partition of one disk).
If you are creating metadevices (e.g., mirrors) on two disks,
particularly if these are boot disks, use instead (for example):
metadb -a -f -c 2 c0t0d0s7 c0t1d0s7
which creates two metadatabases on slice 7 of two disks. For added
safety, in this case (boot disks, even number of metadatabases), add the
following to /etc/system:
set md:mirrored_root_flag=1
As explained above, this will enable normal functionality even should
one disk completely fail (loss of exactly half of extant metadatabases).
If you are simply creating additional metadatabases, as on another
disk or slice, use (for example):
metadb -a c2t0d0s7 # 1 metadatabase on one disk.
metadb -a -c 2 c2t0d0s7 # 2 metadatabases on one disk.
metadb -a -c 2 c2t0d0s7 c2t2d0s7 # 2 metadatabases on each of 2 disks.
Remember that you cannot add metadatabases to a partition already
possessing metadatabases; you must delete existing metadatabases first.
deleting metadatabases
----------------------
You may want to delete metadatabases for several reasons:
- to completely disable DiskSuite (temporarily or permanently)
- to replace a corrupted metadatabase
- to create a new number of metadatabases on a partition
Do so with (for example):
metadb -d [-f] c0t0d0s7
The 'f' flag is necessary when the deletion will result in fewer
metadatabases than recommended (usually just temporarily). Note that
deletion is an all-or-nothing operation; you cannot delete just some of
the existing metadatabases on a disk partition.
checking metadatabases
----------------------
Check metadatabase status with:
metadb [-i]
The 'i' (inquire) flag provides a key to the status characters shown in
the normal output of this command. Note that the use of uppercase in
status characters signals abnormal status.
See the troubleshooting section below if:
- If half or more of the metadatabases have become stale.
- The metadatabases have been totally lost.
making mirrors (p. 71)
======================
There are basically two procedures, depending upon whether or not the
file system to be mirrored can be unmounted. In addition, two extra
steps are required when making the root mirror, which are shown in the
general procedure for making mirrors on unmountable file systems.
Notes
-----
- If creating new metadevices, you must first create metadatabases; see
metadatabases section above.
- Actual mirroring begins when the replica submirrors are synchronized
to the master submirror, which is triggered by the metattach command
in the following procedures. For best results, allow each sync to run
to completion before proceeding. Otherwise, the entire process will
take about three times longer, and will be harder on the disk drives.
Monitor sync'ing progress with metastat (shown in procedures below).
- Creating a two-way mirror, as in this example, involves creating a
mirror metadevice, comprised of two (separate) submirrors: a master
submirror, which is the original mounted slice to mirror; and a slave
(replica) submirror, which is an unmounted slice of equal or greater
size. The mounted slice to mirror is used in the argument to the '-f'
(force) option for metainit, when creating the mirror, and again with
the '-f' option for metadetach, when breaking a mirror. Note that the
replica submirror always syncs to the master submirror, and that this
sync'ing will occur during the boot sequence in the event DiskSuite
detects they are out of sync.
making mirrors of file systems which may be unmounted (/export, /var)
------------------------------------------------------------------------
Example for making mirror of /var:
# '-f' is used with mounted (primary) disk:
metainit -f d24 1 1 c0t0d0s4 # c0t0d0 is primary bootdisk.
metainit d34 1 1 c0t1d0s4 # c0t1d0 is secondary bootdisk.
metainit d14 -m d24 # Put master submirror on primary bootdisk
umount -f /var # Next: mkdir /var/tmp (for vi) if absent.
vi /etc/vfstab # Edit according to sample vfstabs above.
mount /var # Next step attaches replica submirror to
metattach d14 d34 # master submirror, which will cause the
metastat d14 # replica to start sync'ing to master.
Output of metastat for successful creation of /var two-way mirror:
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
d14: Mirror
Submirror 0: d24
State: Okay
Submirror 1: d34
State: Okay
Pass: 1
Read option: roundrobin (default)
Write option: parallel (default)
Size: 10489680 blocks
d24: Submirror of d14
State: Okay
Size: 10489680 blocks
Stripe 0:
Device Start Block Dbase State Hot Spare
c0t0d0s4 0 No Okay
d34: Submirror of d14
State: Okay
Size: 10489680 blocks
Stripe 0:
Device Start Block Dbase State Hot Spare
c0t1d0s4 0 No Okay
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
The key here is "State: Okay". If things are not OK, an error message
would be displayed, along with advice for correcting. Each hour the
MetaCheck script runs out of sysmon's crontab, which among other checks,
checks that these states are all 'Okay'; if not, a message is sent to
'oncall'.
making mirrors of file systems which cannot be unmounted (/,/usr,/swap)
-----------------------------------------------------------------------
Example for making mirror of /:
# '-f' is used with mounted (master) disk:
metainit -f d20 1 1 c0t0d0s0 # c0t0d0 is primary bootdisk.
metainit d30 1 1 c0t1d0s0 # c0t1d0 is secondary bootdisk.
metainit d10 -m d20 # Put master submirror on primary bootdisk
If making the root mirror (as in this example) do this:
metaroot d10 # Alters /etc/system and /etc/vfstab; clean
lockfs -af # up vfstab (if desired) and comment out w/
# '*' the following in the new /etc/system:
# forceload: misc/md_trans
# forceload: misc/md_raid
# forceload: misc/md_hotspares
# forceload: misc/md_sp
else:
vi /etc/vfstab # Edit according to sample vfstabs above.
init 6 # Next step attaches replica submirror to
metattach d10 d30 # master submirror, which will cause the
metastat d10 # replica to start sync'ing to master.
breaking mirrors (p. 182)
=========================
There are basically two procedures, depending upon whether or not the
mirrored file system can be unmounted. In addition, an extra step is
required when breaking the root mirror, which is shown in the general
procedure for breaking mirrors on unmountable file systems.
breaking mirrors of file systems which may be unmounted (/export, /var,)
------------------------------------------------------------------------
Example for breaking mirror of /var:
metastat d14 # All d?4 devices should be present and 'Okay'.
metadetach -f d14 d24 # Detaches the master mirror, which represents
umount -f /var # the original mounted disk to mirror. Change the
vi /etc/vfstab # line in vfstab which uses the mirror metadevice
mount /var # to use instead the actual device path.
metaclear -r d14 # Clears d14 and d34.
metaclear d24 # Clears d24 (previously detached master mirror).
metastat d14 # Should see "unit not set up" and no d?4 devices.
breaking mirrors of filesystems which cannot be unmounted (/,/usr,/swap)
------------------------------------------------------------------------
Example for breaking mirror of /:
metastat d10 # All d?0 devices should be present and 'Okay'.
# If breaking the root mirror (as in this example) do this:
metaroot /dev/dsk/c0t0d0s0 # Alters /etc/system and /etc/vfstab; clean
# up (if desired) and update SunFiles.
# else do this:
vi /etc/vfstab # Change the vfstab line specifying the mirror
# metadevice to instead use the actual device path
# Next:
metadetach -f d10 d20 # Detaches the master mirror, which represents
# the original mounted disk to mirror.
# Before rebooting (next step) sanity check /etc/system and /etc/vfstab.
# For /etc/system, make sure there is no uncommented 'rootdev' SDS
# setting. For /etc/vfstab, make sure the device paths for '/' no longer
# reference metadevices. System will not boot properly if metaroot did
# not properly update these files.
init 6
metaclear -r d10 # Clears d10 and d30.
metaclear d20 # Clears d20 (previously detached).
metastat d10 # You should see no d?0 devices.
other mirroring tasks
=====================
Taking a submirror offline
--------------------------
When a submirror is metaoffline'd, DiskSuite halts all I/O to that
submirror. Writes that would have occurred are instead journaled, so
that when the submirror is later metaonline'd, sync'ing occurs quickly
as only those journaled writes are executed. Taking a submirror offline
may be desirable for quiescing a submirror prior to backing it up, or
when it is is known that the submirror device will become temporarily
inaccessible, as in taking down a SCSI chain. Do not reboot after taking
a submirror offline; instead, you will need to detach the submirror
fully (see next section). This procedure is only for temporarily
offlining a submirror, and only then when the offlined submirror will
not be written to. Example (for one partition):
metaoffline d16 d36
This command detaches the (secondary) submirror of partition 6. If you
will be offlining an entire disk (usual case), you must issue this
command for each partition mirrored.
To bring a submirror back online, use (for example):
metaonline d16 d36
Note that the metaonline command can only be issued when the submirror
has been previously metaoffline'd; without having been metaoffline'd, no
journal is available for fast syncing.
Detaching a submirror
---------------------
metadetach is useful when a disk will be removed and replaced with a
different disk (or will be so changed that it will be essentially a
different disk), or when it is desired to preserve its data from
updates (such as when applying patch clusters). Once returned to the
mirror it may be metattach'd for complete resyncing.
metadetach -f d16 d36
To reattach the above detached submirror:
metattach d16 d36 # Note that data is written _from_ d16 _to_ d36.
Replacing a failed disk
-----------------------
metareplace allows the substitution of a good disk slice of equal or
greater size for a failed disk slice. It is also used to reenable a
disk slice which has been marked as needing maintenance, which will
happen if the associated disk loses power. In this case there is only
one argument to the metareplace command, as the disk slice is
'replacing' itself (e.g.):
metareplace -e d10 c0t0d0s0
Create a third submirror for use in cloning to fourth disk
----------------------------------------------------------
(Following are commands used in procedure performed on spike:)
on both unused disks:
# delete any existing metadatabases
sudo metadb -d c0t0d0s7 c1t0d0s7
# repartition (use 18GB6800 in spike:/etc/format.dat)
sudo format
to create new (third) mirror disk:
# make file systems:
for slice in c1t0d0s0 c1t0d0s1 c1t0d0s3 c1t0d0s4 c1t0d0s5
do
sudo newfs /dev/rdsk/$slice
sudo fsck -y /dev/rdsk/$slice
done
# create new metadatabases:
sudo metadb -a -f -c 3 c1t0d0s7
# create new (third) submirror (of each file system):
# /
metastat d40
sudo metainit d70 1 1 c1t0d0s0
sudo metattach d40 d70
while [ 1 ]; do metastat d40; sleep 5; done
# /usr
metastat d41
sudo metainit d71 1 1 c1t0d0s1
sudo metattach d41 d71
while [ 1 ]; do metastat d41; sleep 5; done
# /var
metastat d43
sudo metainit d73 1 1 c1t0d0s3
sudo metattach d43 d73
while [ 1 ]; do metastat d43; sleep 5; done
# /export/local
metastat d44
sudo metainit d74 1 1 c1t0d0s4
sudo metattach d44 d74
while [ 1 ]; do metastat d44; sleep 5; done
# /export/home
metastat d45
sudo metainit d75 1 1 c1t0d0s5
sudo metattach d45 d75
while [ 1 ]; do metastat d45; sleep 5; done
# swap
metastat d46
sudo metainit d76 1 1 c1t0d0s6
sudo metattach d46 d76
while [ 1 ]; do metastat d46; sleep 5; done
to create new clone disk:
# Prerequisites:
# /etc/system.non-sds
# /etc/vfstab.non-sds
# /usr/local/adm/bin/clone.sds
# /usr/software/adm/sbin/DSfreeze
# /usr/software/adm/sbin/FSfreeze
# quiesce third mirror disk # Takes a minute or so ... wait 'til completed.
sudo /usr/software/adm/sbin/DSfreeze 4 7 3000 # offlines d7x for 50 minutes
# clone third mirror disk
sudo /usr/local/adm/bin/clone.sds c1t0 c0t0 # makes the clone independently
# bootable (non-SDS)
# add to root crontab:
5 3 * * 0 /usr/software/adm/sbin/DSfreeze 4 7 3000
10 3 * * 0 /usr/local/adm/bin/clone.sds c1t0 c0t0
Maintaining a backup image during software upgrades (reversible upgrade):
-------------------------------------------------------------------------
In this example, the krb software on kdc1 is being upgraded. In case of
problems with the upgrade, it may become necessary to back out to the former
krb software. It is thus desired to keep the old (current) image on one disk,
while performing the upgrade on the other disk, with the capacity, after the
upgrade, to resync the system to one disk or the other. Following is one
solution:
Prior to software upgrade:
# for i in 0 1 3 4 5 6
> do
> metadetach d1$i d3$i
> done
At this point disk1 maintains an image of the system prior to upgrade. If it
becomes necessary to fall back to this prior image, then:
# mount /dev/dsk/c0t1d0s0 /export/mnt
# cd /export/mnt
# cp -p system.non-sds system
# cp -p vfstab.non-sds vfstab
# vi vfstab (s/c0t0/c0t1/g)
# cd
# umount /export/mnt
# for i in 0 1 3 4 5
> do
> fsck -y -o f /dev/rdsk/c0t1d0s$i
> done
# init 0
ok> boot disk1
If backing out to c0t1 is not necessary, then at some point the
submirrors should be reattached:
# for i in 0 1 3 4 5 6
> do
> metattach d1$i d3$i
> done
Note that in the first case (backout to disk1) followup work would involve
reestablishing the mirrors, making c0t0 again the primary submirror.
troubleshooting
===============
- stale databases (p. 271): this is the term used when a quorum (>50%)
of databases are unusable. If you see this term, you are likely having
troubles booting, and are being prompted to continue in single user
mode for repairs. So, go into single user, and use metadb -i for a
listing of databases and their attributes. The presence of attributes
represented with an uppercase letter generally indicates a problem,
although problems can still exist when only lowercase letters appear.
The solution is to delete all problem databases, then reboot and
recreate them. Sample output of metadb -i (with stale databases):
flags first blk block count
a m pc lu 16 1034 /dev/dsk/c0t0d0s7
a pc lu 1050 1034 /dev/dsk/c0t0d0s7
a pc lu 2084 1034 /dev/dsk/c0t0d0s7
a p u 16 1034 /dev/dsk/c0t1d0s7
a p 1050 1034 /dev/dsk/c0t1d0s7
a p u 2084 1034 /dev/dsk/c0t1d0s7
o - replica active prior to last mddb configuration change
u - replica is up to date
l - locator for this replica was read successfully
c - replica's location was in /etc/lvm/mddb.cf
p - replica's location was patched in kernel
m - replica is master, this is replica selected as input
W - replica has device write errors
a - replica is active, commits are occurring to this replica
M - replica had problem with master blocks
D - replica had problem with data blocks
F - replica had format problems
S - replica is too small to hold current data base
R - replica had device read errors
Here we see six replicas, three each on slice 7 of disks 0 and 1.
Problems exist for those on the second disk. So, delete them with:
metadb -f -d c0t1d0s7
This will attempt to delete the stale databases, which may not be
possible if the stale databases reside on a failed disk. But it will
update the remaining good databases to remove references to the stale
databases. It will also attempt to update /etc/lvm/mddb.cf, which
attempt will fail, as the root file system at this point is read-only;
ignore this warning. This file will be updated later when you recreate
the databases.
If the databases went bad because the disk went bad:
init 5 # Power off, replace disk, then power back on and reboot.
else:
init 6 # Reboot.
Recreate deleted databases with (for example):
metadb -a -c 3 c0t1d0s7
The '-c 3' says to create three replicas on the indicated slice.
- lost databases (p. 261): If you have access to a copy of
/etc/lvm/md.cf (ideally in SunFiles), and have not altered metadevice
configuration (i.e., creating/deleting/revising metadevices) since the
loss of the metadatabases, then perform the following:
# Recreate (empty) databases. See procedure above in 'metadatabases'.
cd /etc/lvm
cp -p md.tab md.tab. # Back up (however) existing md.tab.
cp -p md.cf md.tab
metainit -n -a # '-a' specifies using md.tab; '-n' just checks syntax.
metainit -a # Assuming syntax is correct, commit to reinitializing.
# Re-attach replica submirrors (see "making mirrors" section above).
- invalid vfstab at boot: an invalid vfstab will cause an attempted boot
to halt at single user mode, with error messages such as "not this
fstype" or "failed write". To fix, in single user mode, fsck the
correct root device, remount it read/write, run metaroot, then reboot.
For example:
# fsck /dev/md/rdsk/d10
# mount -o rw,remount /dev/md/dsk/d10
# metaroot d10
# init 6