RHEL: /data Disappeared After Reboot? Troubleshoot LVM and Multipath Before Touching pvcreate
RHEL: /data Disappeared After Reboot? Troubleshoot LVM and Multipath Before Touching pvcreate
RHEL：重启后 /data 消失了？在执行 pvcreate 之前，请先排查 LVM 和多路径 (Multipath)
A filesystem disappearing after a reboot is one of those incidents where a quick “fix” can make the situation much worse. Imagine this: /data is missing, Application cannot start, systemd reports a mount failure, Server may even enter emergency mode. The first reaction might be: pvcreate, vgcreate, lvcreate, mkfs.xfs. On an existing production LUN, that can be exactly the wrong thing to do. The problem may not be LVM at all.
文件系统在重启后消失是那种“快速修复”反而可能让情况变得更糟的事故之一。想象一下：/data 丢失了，应用程序无法启动，systemd 报告挂载失败，服务器甚至可能进入紧急模式。第一反应可能是执行：pvcreate、vgcreate、lvcreate、mkfs.xfs。但在现有的生产环境 LUN 上，这恰恰是错误的做法。问题可能根本不在 LVM。
Start with the failure layers
从故障层级开始排查
In an enterprise RHEL environment, /data can depend on several layers: /data ↓ Filesystem ↓ Logical Volume ↓ Volume Group ↓ Physical Volume ↓ Multipath ↓ SAN LUN ↓ Storage paths. The goal is simple: Find the first broken layer before changing anything.
在企业级 RHEL 环境中，/data 可能依赖于多个层级： /data ↓ 文件系统 ↓ 逻辑卷 (Logical Volume) ↓ 卷组 (Volume Group) ↓ 物理卷 (Physical Volume) ↓ 多路径 (Multipath) ↓ SAN LUN ↓ 存储路径。 目标很简单：在进行任何更改之前，找到第一个损坏的层级。
1. Is /data actually expected to be mounted?
Start with: findmnt /data, grep -w '/data' /etc/fstab, lsblk -f.
Then inspect the boot errors: journalctl -b -p err..alert --no-pager.
You may see something like: Failed to mount /data, Dependency failed for Local File Systems. That only proves the mount failed. It does not prove that the filesystem is corrupted.
1. /data 是否确实应该被挂载？
首先执行：findmnt /data、grep -w '/data' /etc/fstab、lsblk -f。
然后检查启动错误：journalctl -b -p err..alert --no-pager。
你可能会看到类似这样的信息：Failed to mount /data，Dependency failed for Local File Systems。这只能证明挂载失败，并不能证明文件系统已损坏。
2. Does the Logical Volume exist?
Check LVM: lvs -a -o lv_name,vg_name,lv_attr,devices.
If the LV is missing, move down one layer. Do not recreate it yet.
2. 逻辑卷是否存在？
检查 LVM：lvs -a -o lv_name,vg_name,lv_attr,devices。
如果 LV 丢失，请向下排查一层。切勿立即重新创建它。
3. Does the Volume Group exist?
vgs. If vg_data is missing, resist the temptation to run vgcreate. Check the PV first.
3. 卷组是否存在？
执行 vgs。如果 vg_data 丢失，请抵制住运行 vgcreate 的冲动。先检查物理卷 (PV)。
4. Is the Physical Volume visible?
pvs -o pv_name,vg_name,pv_size,pv_free.
If /dev/mapper/mpatha is missing, the problem is probably below LVM. Now the investigation changes direction.
4. 物理卷是否可见？
执行 pvs -o pv_name,vg_name,pv_size,pv_free。
如果 /dev/mapper/mpatha 丢失，问题很可能在 LVM 之下。现在调查方向需要改变了。
5. Check Multipath
multipath -ll. You can inspect the individual paths with: multipathd show paths and lsscsi.
If the expected WWID is absent completely, recreating the VG will not fix the root cause. The SAN LUN may simply not be visible to the server.
5. 检查多路径 (Multipath)
执行 multipath -ll。你可以通过 multipathd show paths 和 lsscsi 来检查各个路径。
如果预期的 WWID 完全不存在，重新创建 VG 并不能解决根本原因。SAN LUN 可能根本没有被服务器识别到。
A typical failure chain
典型的故障链
Suppose your checks give this result:
findmnt /data → not mounted
lvs → lv_data missing
vgs → vg_data missing
pvs → expected PV missing
multipath -ll → expected WWID missing
Now the picture is very different: SAN LUN not visible ↓ Multipath map missing ↓ PV missing ↓ VG missing ↓ LV missing ↓ Filesystem unavailable ↓ /data not mounted. The missing filesystem was only the visible symptom. The real failure occurred much lower in the stack.
假设你的检查结果如下：
findmnt /data → 未挂载
lvs → lv_data 丢失
vgs → vg_data 丢失
pvs → 预期的 PV 丢失
multipath -ll → 预期的 WWID 丢失
现在情况完全不同了： SAN LUN 不可见 ↓ 多路径映射丢失 ↓ PV 丢失 ↓ VG 丢失 ↓ LV 丢失 ↓ 文件系统不可用 ↓ /data 未挂载。 文件系统丢失只是表面症状，真正的故障发生在堆栈的更底层。
Commands I would avoid at this stage
我在现阶段会避免使用的命令
Unless you have proven that the storage is new and intentionally being initialized, avoid commands such as: pvcreate, vgcreate, lvcreate, mkfs.xfs.
These are creation commands, not troubleshooting commands. On an existing production disk, they may overwrite metadata you are trying to recover.
除非你已确认存储是全新的且有意进行初始化，否则请避免使用以下命令：pvcreate、vgcreate、lvcreate、mkfs.xfs。
这些是创建命令，而非排查命令。在现有的生产磁盘上，它们可能会覆盖你试图恢复的元数据。
Evidence to collect before escalation
升级前需要收集的证据
Before calling the storage, SAN, VMware or infrastructure team, collect evidence:
date, hostnamectl, uname -r, lsblk -f, findmnt, pvs, vgs, lvs -a -o +devices, multipath -ll, multipathd show paths, journalctl -b -p err..alert --no-pager.
In complex cases, and if permitted by your environment: sos report. This gives the next support level something concrete to work with.
在联系存储、SAN、VMware 或基础设施团队之前，请收集以下证据：
date、hostnamectl、uname -r、lsblk -f、findmnt、pvs、vgs、lvs -a -o +devices、multipath -ll、multipathd show paths、journalctl -b -p err..alert --no-pager。
在复杂情况下，如果环境允许，请运行 sos report。这能为下一级支持提供具体的分析依据。
The troubleshooting habit that matters
关键的排查习惯
When /data disappears, don’t immediately ask: “Which command can recreate /data?” Ask: “Which command proves why /data disappeared?” That small change in mindset can prevent a recoverable infrastructure incident from becoming a data-loss incident.
当 /data 消失时，不要立即问：“哪个命令可以重建 /data？” 而要问：“哪个命令能证明 /data 为什么消失？” 这种思维方式的微小转变，可以防止一次可恢复的基础设施事故演变成数据丢失事故。
(Resource links omitted for brevity) (资源链接略)
If you work on RHEL production systems, I’d be interested to know: What is the most dangerous “quick fix” you’ve seen during a storage incident?
如果你从事 RHEL 生产系统工作，我很想知道：你在存储事故中见过的最危险的“快速修复”是什么？