Skip to content

[Bug]: shim may re-format dstack-managed volumes #3973

@r4victor

Description

@r4victor

Steps to reproduce

This is time-dependent and cannot be easily reproduced.

shim creates a file system on dstack-managed volumes when the volume is attached for the first time. Before the formatting, it checks if the filesystem already exists on the volume via lsblk

// Run the lsblk command to get filesystem type
cmd := exec.CommandContext(ctx, "lsblk", "-no", "FSTYPE", deviceName)
var out bytes.Buffer
cmd.Stdout = &out
if err := cmd.Run(); err != nil {
return false, fmt.Errorf("failed to check if disk is formatted: %w", err)
}

The issue is that lsblk cannot be used reliably to check for filesystem existence like that. "The lsblk command reads the sysfs filesystem and udev db to gather information." – the information about the filesystem may not be available yet. If that happens, dstack will reformat the volume, leading to data loss.

Actual behaviour

No response

Expected behaviour

Consider using blkid -p -o value -s TYPE <dev>. as blkid can read volume superblock directly.

dstack version

master

Server logs

Additional information

No response

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingmajor

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions