It’s important to understand the volume chains for troubleshooting snapshot create/delete issues. This post will explain the correlation of volume chains.

Volume chains in both Engine Database and Host need to be checked as you need to check if the REAL status of the disks is the same as what you see in the database. The procedure shown here is for LVM based block storage (iSCSI and FiberChannel). For NFS it’s much simpler, just look for files with the image id as a name. When we create a snapshot, RHV “freezes” the base disk and creates a copy on the write (COW) layer on top to store the changes. The COW layer is implemented as a qcow snapshot, that stores a reference to the base image that contains the original disk (or previous layer).

1. Check from Engine database:

– The ‘first’ snapshot for VMs is ‘Active VM’ in RHV. ‘Active VM’ is not an actual snapshot but just shows the “current status” as a snapshot. It is the main disk (parentid zero). For example:

vm_name | snapshot_name | snapshot_status | image_guid | image_group_id | parentid | imagestatus
------------+----------------------+-----------------+--------------------------------------+--------------------------------------+--------------------------------------+-------------
TestVM | Active VM | OK | 4ccf601e-ec11-4c10-be8c-3b1a6e53aa51 | 9f13d2e3-eed5-4af2-936c-358bc4948608 | 00000000-0000-0000-0000-000000000000 | 1
(1 row)

– If you create a snapshot, then the real first snapshot will be created. This first snapshot is a COW snapshot that uses the previous disk as base image. The parentid is zero. After the first snapshot is taken, the ‘Active VM’ parentid becomes the first snapshot’s image_guid.

Example: the first snapshot name is volumechain1. From the below DB output, you can see the image_guid and parentid relationship:

vm_name | snapshot_name | snapshot_status | image_guid | image_group_id | parentid | imagestatus
------------+---------------+-----------------+--------------------------------------+--------------------------------------+--------------------------------------+-------------
TestVM | Active VM | OK | 3fdf455e-52e6-48da-81f5-475cad796d21 | 9f13d2e3-eed5-4af2-936c-358bc4948608 | 4ccf601e-ec11-4c10-be8c-3b1a6e53aa51 | 1
TestVM | volumechain1 | OK | 4ccf601e-ec11-4c10-be8c-3b1a6e53aa51 | 9f13d2e3-eed5-4af2-936c-358bc4948608 | 00000000-0000-0000-0000-000000000000 | 1
(2 rows)

Example for the second(volumechain2)/third(volumechain3) snapshots. You can see that the second snapshot’s parentid is the first snapshot volumeid and the third snapshot’s parentid is the second snapshot volume_id.

vm_name | snapshot_name | snapshot_status | image_guid | image_group_id | parentid | imagestatus
------------+---------------+-----------------+--------------------------------------+--------------------------------------+--------------------------------------+-------------
TestVM | Active VM | OK | 043dfd54-30d2-4437-9cba-2eded92136b6 | 9f13d2e3-eed5-4af2-936c-358bc4948608 | 3fdf455e-52e6-48da-81f5-475cad796d21 | 1
TestVM | volumechain1 | OK | 4ccf601e-ec11-4c10-be8c-3b1a6e53aa51 | 9f13d2e3-eed5-4af2-936c-358bc4948608 | 00000000-0000-0000-0000-000000000000 | 1
TestVM | volumechain2 | OK | 3fdf455e-52e6-48da-81f5-475cad796d21 | 9f13d2e3-eed5-4af2-936c-358bc4948608 | 4ccf601e-ec11-4c10-be8c-3b1a6e53aa51 | 1
(3 rows)
vm_name | snapshot_name | snapshot_status | image_guid | image_group_id | parentid | imagestatus
------------+---------------+-----------------+--------------------------------------+--------------------------------------+--------------------------------------+-------------
TestVM | Active VM | OK | c9a717e4-bc90-4ef5-900d-777bf01b43bf | 9f13d2e3-eed5-4af2-936c-358bc4948608 | 043dfd54-30d2-4437-9cba-2eded92136b6 | 1
TestVM | volumechain1 | OK | 4ccf601e-ec11-4c10-be8c-3b1a6e53aa51 | 9f13d2e3-eed5-4af2-936c-358bc4948608 | 00000000-0000-0000-0000-000000000000 | 1
TestVM | volumechain2 | OK | 3fdf455e-52e6-48da-81f5-475cad796d21 | 9f13d2e3-eed5-4af2-936c-358bc4948608 | 4ccf601e-ec11-4c10-be8c-3b1a6e53aa51 | 1
TestVM | volumenchain3 | OK | 043dfd54-30d2-4437-9cba-2eded92136b6 | 9f13d2e3-eed5-4af2-936c-358bc4948608 | 3fdf455e-52e6-48da-81f5-475cad796d21 | 1
(4 rows)

2. Check from Host:

– Check the domblk info:

# virsh -r domblklist TestVM
Target Source
---------------------------------------------------------------------------------------------------------------------------------------------------------------
hdc -
sda /rhev/data-center/mnt/blockSD/c95e9f3e-79cb-47ab-9825-8093ee12e42b/images/9f13d2e3-eed5-4af2-936c-358bc4948608/c9a717e4-bc90-4ef5-900d-777bf01b43bf 

– Check LV tags: The LVM backend stores the data in Logical Volumes (LV). RHV tags all LV with the disk id that uses that LV. From the below command LV tags, you can see the volume chains.

# lvs -o +tags|grep 9f13d2e3-eed5-4af2-936c-358bc4948608
043dfd54-30d2-4437-9cba-2eded92136b6 c95e9f3e-79cb-47ab-9825-8093ee12e42b -wi-ao---- 1.00g IU_9f13d2e3-eed5-4af2-936c-358bc4948608,MD_8,PU_3fdf455e-52e6-48da-81f5-475cad796d21 >>>> third snapshot
3fdf455e-52e6-48da-81f5-475cad796d21 c95e9f3e-79cb-47ab-9825-8093ee12e42b -wi-ao---- 1.00g IU_9f13d2e3-eed5-4af2-936c-358bc4948608,MD_6,PU_4ccf601e-ec11-4c10-be8c-3b1a6e53aa51 >>>>> second snapshot
4ccf601e-ec11-4c10-be8c-3b1a6e53aa51 c95e9f3e-79cb-47ab-9825-8093ee12e42b -wi-ao---- 48.00g IU_9f13d2e3-eed5-4af2-936c-358bc4948608,MD_3,PU_00000000-0000-0000-0000-000000000000 >>>>> first snapshot
c9a717e4-bc90-4ef5-900d-777bf01b43bf c95e9f3e-79cb-47ab-9825-8093ee12e42b -wi-ao---- 1.00g IU_9f13d2e3-eed5-4af2-936c-358bc4948608,MD_11,PU_043dfd54-30d2-4437-9cba-2eded92136b6 >>>>> Active VM 

– Check the qemu images. For example:

# qemu-img info /dev/c95e9f3e-79cb-47ab-9825-8093ee12e42b/4ccf601e-ec11-4c10-be8c-3b1a6e53aa51
image: /dev/c95e9f3e-79cb-47ab-9825-8093ee12e42b/4ccf601e-ec11-4c10-be8c-3b1a6e53aa51
file format: raw
virtual size: 48G (51539607552 bytes) >>>>>>>>>>>
disk size: 0

Dump volume chains. For example:

# vdsm-tool dump-volume-chains c95e9f3e-79cb-47ab-9825-8093ee12e42b |grep -A14 9f13d2e3-eed5-4af2-936c-358bc4948608
image: 9f13d2e3-eed5-4af2-936c-358bc4948608
- 4ccf601e-ec11-4c10-be8c-3b1a6e53aa51             >>>>this is the first snapshot volume_id
status: OK, voltype: INTERNAL, format: RAW, legality: LEGAL, type: PREALLOCATED, capacity: 51539607552, truesize: 51539607552
- 3fdf455e-52e6-48da-81f5-475cad796d21             >>>>this is the second snapshot volume_id
status: OK, voltype: INTERNAL, format: COW, legality: LEGAL, type: SPARSE, capacity: 51539607552, truesize: 1073741824
- 043dfd54-30d2-4437-9cba-2eded92136b6             >>>>this is the third snapshot volume_id
status: OK, voltype: INTERNAL, format: COW, legality: LEGAL, type: SPARSE, capacity: 51539607552, truesize: 1073741824
- c9a717e4-bc90-4ef5-900d-777bf01b43bf             >>>>this is Active VM Leaf volume_id
status: OK, voltype: LEAF, format: COW, legality: LEGAL, type: SPARSE, capacity: 51539607552, truesize: 1073741824