fsck fails on file partitions

[Nahor <nahor.j+openrc@gmail.com>]

I have a file that I mount as a partition.
From /etc/fstab:
/usr/portage_bin/ebuildPartition /usr/portage ext2 loop,noatime,nodev 0 2

When the fsck service runs at the boot, it fails because the filesystem hosting that file is readonly:

fsck |Reiserfs super block in block 16 on 0x803 of format 3.6 with standard journal
fsck |Blocks (total/free): 5114688/2228115 by 4096 bytes
fsck |Filesystem is clean
fsck |Filesystem seems mounted read-only. Skipping journal replay.
fsck |Checking internal tree..finished
fsck |fsck.ext2: Read-only file system while trying to open /usr/portage_bin/ebuildPartition
fsck |Disk write-protected; use the -n option to do a read-only
fsck |check of the device.
fsck | * Operational error

[!!]

root | * Remounting root filesystem read/write ...

[ok]

[roy]

This makes perfect sense when you think about it - you only need to fsck the filesystem where the loopback file resides, in your case /usr (or / if /usr is on /)

So set the passno to 0 so fsck skips it.

If you really need to fsck it, you'll need to set it noauto and re-open this bug so we can talk about adding a new init script to fsck and mount loopback files.

[Nahor <nahor.j+openrc@gmail.com>]

Why would I only need to fsck the hosting filesystem? Unless there is a data journaling, there is nothing to guarantee that the data will not be corrupted.

Shouldn't the script then run fsck in readonly mode then run a maintenance shell if a corruption is found instead of allowing the system to run on a potentially corrupted filesystem?

Or even better, couldn't fsck run on the loopback filesystems after the hosting filesystem has become writable?

[roy]

(In reply to comment #2)

Or even better, couldn't fsck run on the loopback filesystems after the hosting
filesystem has become writable?

You could have this chain.

fsck /
fsck /usr
mount /
mount /usr
--- we work up until here ---
fsck /usr/loop1
mount /usr/loop1
fsck /usr/loop1/loop2
mount /usr/loop1/loop2

The fsck and localmount scripts aren't parallised with each other which means we would have to call fsck and mount many times in a new init script. You would still have to set the passno to 0 for the loopback filesystems, and possibly noauto as well as we would need to skip it in the normal localmount script.

Do you know of an OS where your wanted setup works for reference?

[Sven <skoehler@upb.de>]

There are two easy algorithms, that make the thing work again:
(it used to work with baselayout only because it mounted the rootfs read/write before fsck'ed mounted the other ones)

algo1:

for each filesystem do

fsck filesystem
mount filesystem

done

maintain the order specified in the fstab

algo2:

for each non-loop filesystem do

fsck filesystem

done
for each non-loop filesystem do

mount filesystem

done
for each loop filesystem do

fsck filesystem

done
for each loop filesystem do

mount filesystem

done

So of course, filesystems using loop devices should be fsck'ed. Yet, dependency tracking is difficult. So i think, algo1 is pretty easy - although fsck and mount init.d scripts would have to be merged. Are there any objections to algo1?

So algo2 needs two more init.d scripts to seperate fsck'ing/mounting of loop filesystems from the other normal filesystems. But i it's less general than algo1. Aren't there more situation, where we have dependencies? And in algo1, it's the user's task to resolve dependencies by setting a proper order in fstab.

[roy]

(In reply to comment #4)

There are two easy algorithms, that make the thing work again:
(it used to work with baselayout only because it mounted the rootfs read/write
before fsck'ed mounted the other ones)

That only works if the loops are on /
It's a lot more common to have a small / and /usr seperate (like say on an lvm volume) than it is to have what we're talking about here.

So of course, filesystems using loop devices should be fsck'ed. Yet, dependency
tracking is difficult. So i think, algo1 is pretty easy - although fsck and
mount init.d scripts would have to be merged. Are there any objections to
algo1?

Yes. The fsck, root and localmount init scripts are designed to let fsck and mount walk through fstab at their leisure. What you are proposing breaks this.

So algo2 needs two more init.d scripts to seperate fsck'ing/mounting of loop
filesystems from the other normal filesystems. But i it's less general than
algo1. Aren't there more situation, where we have dependencies? And in algo1,
it's the user's task to resolve dependencies by setting a proper order in
fstab.

No, we just need another init script to do something like so

for loop in $(fstabinfo --option loop | sort); do

if fsck ${fsck_args--p} "${loop}"; then

mount "${loop}"

fi

done

You'll still need to set the passno to 0 and noauto in fstab for each loop.
Also, fstabinfo cannot extract by option right now, so that will need to be coded too.

[Sven <skoehler@upb.de>]

So of course, filesystems using loop devices should be fsck'ed. Yet, dependency
tracking is difficult. So i think, algo1 is pretty easy - although fsck and
mount init.d scripts would have to be merged. Are there any objections to
algo1?

Yes. The fsck, root and localmount init scripts are designed to let fsck and
mount walk through fstab at their leisure. What you are proposing breaks this.

Yes, indeed. I was aware of that. It's so confusing, that there are two different sorting orders: one for fsck, one for mount.

So we better maintain fstab semantics, and withdraw algo1.

No, we just need another init script to do something like so

for loop in $(fstabinfo --option loop | sort); do

if fsck ${fsck_args--p} "${loop}"; then

mount "${loop}"

fi

done

You'll still need to set the passno to 0 and noauto in fstab for each loop.
Also, fstabinfo cannot extract by option right now, so that will need to be
coded too.

Are you proposing to automount filesystems that are marked as noauto?
Are you proposing to fsck filesystems with passno 0?

Either of it destroyes fstab semantics. AFAIK passno 0 mean: "don't fsck" and noauto means "don't mount it on boot".

So actually, if that's possible, extend fstabinfo to be abled to:

• output a list of non-loop devies in passno order
• output a list of non-loop devies in mount order
• output a list of loop devices in passno ordner
• output a list of loop devices in mount ordner

And then handle each filesystems in these 4 list accordingly.

[roy]

(In reply to comment #6)

Are you proposing to automount filesystems that are marked as noauto?
Are you proposing to fsck filesystems with passno 0?

Either of it destroyes fstab semantics. AFAIK passno 0 mean: "don't fsck" and
noauto means "don't mount it on boot".

0 means don't fsck on boot
noauto means don't mount on boot

To clarify:-
If fsck is called without any arguments it then fscks the disks according to passno. It will do this in parallel if possible.
If mount is called with -a then it mounts all disks except those specified by fs type exclusions and the noauto keyword.
You can read the fstab manpage to verify this.

This is the default for our scripts, and for pretty much every *NIX out there.

I've had a good long look for loopback fstab examples. I can't find any where
passno is not 0 and noauto is not an option. In other words, it looks like the norm to have special init scripts for this. Although most of the examples I found dealt with encryption.

So actually, if that's possible, extend fstabinfo to be abled to:

• output a list of non-loop devies in passno order
• output a list of non-loop devies in mount order
• output a list of loop devices in passno ordner
• output a list of loop devices in mount ordner

And then handle each filesystems in these 4 list accordingly.

This of course means that we have to serialise each operation for fsck and mount which is not exactly efficient.

[Sven <skoehler@upb.de>]

(In reply to comment #7)

I've had a good long look for loopback fstab examples. I can't find any where
passno is not 0 and noauto is not an option. In other words, it looks like the
norm to have special init scripts for this. Although most of the examples I
found dealt with encryption.

I'm not very lucky with it. Let me put it into questions:

How do i specify a loop filesystem, which get's not mounted on boot, if noauto is ignored? How do i specify the fsck order for loop filesystems if passno has to be 0? (I cannot think of any usecase, where passno would be important - but i really can imagine, that there might be loop filesystems in fstab, which should NOT be mounted on boot mixed with other ones, that should be mounted)

So actually, if that's possible, extend fstabinfo to be abled to:

• output a list of non-loop devies in passno order
• output a list of non-loop devies in mount order
• output a list of loop devices in passno ordner
• output a list of loop devices in mount ordner

And then handle each filesystems in these 4 list accordingly.

This of course means that we have to serialise each operation for fsck and
mount which is not exactly efficient.

Yes, that's true and very suboptimal.

I took a look at "fsck --help", and it doesn't seem like it takes multiple devicies in the parameter list. Yet, a quick test showed, that it seems to work. But i don't know, if it would do that in parallel. But i guess, going this way will cause other problems.

[roy]

(In reply to comment #8)

I'm not very lucky with it. Let me put it into questions:

How do i specify a loop filesystem, which get's not mounted on boot, if noauto
is ignored? How do i specify the fsck order for loop filesystems if passno has
to be 0? (I cannot think of any usecase, where passno would be important - but
i really can imagine, that there might be loop filesystems in fstab, which
should NOT be mounted on boot mixed with other ones, that should be mounted)

You cannot do this in /etc/fstab AND be compatible with system default fsck and mount commands, hence the need for configuration in an external file.

I would imagine this in /etc/conf.d/fsckmount
fsckmounts="/usr/loop1 /usr/loop2"

Yes, that's true and very suboptimal.

I took a look at "fsck --help", and it doesn't seem like it takes multiple
devicies in the parameter list. Yet, a quick test showed, that it seems to
work. But i don't know, if it would do that in parallel. But i guess, going
this way will cause other problems.

It does work, but fsck and mount make no distinction between a device and a mount point if a name matches both. For example

/dev/hda2 /usr
/usr /foo bind

Now try and fsck /usr and /foo. Things get even more messy when you add umount into the mix.

[Nahor <nahor.j+openrc@gmail.com>]

The problem I see with having a separate config file is that usually people need to only look at fstab to see what they want to know about the mounts. They may forget, even they even know in the first place about the other file.

This may be a bit overboard, if even possible, but what about using temporary fstab files for each passno?

mount -t tmpfs /tmpfstab
foreach passno in /etc/fstab

build /tmpfstab/fstab with partitions with passno
FSTAB_FILE=/tmpfstab/fstab fsck ...
mount all partition from /tmpfstab/fstab

done
umount /tmpfstab

[roy]

(In reply to comment #10)

The problem I see with having a separate config file is that usually people
need to only look at fstab to see what they want to know about the mounts. They
may forget, even they even know in the first place about the other file.

This may be a bit overboard, if even possible, but what about using temporary
fstab files for each passno?

mount -t tmpfs /tmpfstab
foreach passno in /etc/fstab

build /tmpfstab/fstab with partitions with passno
FSTAB_FILE=/tmpfstab/fstab fsck ...
mount all partition from /tmpfstab/fstab

done
umount /tmpfstab

Nice idea, but FSTAB_FILE would be Linux only. OpenRC also supports FreeBSD and NetBSD out of the box. Also, $RC_SVCDIR is writeable at this point as it's mounted tmpfs.
Now, if people put a comment in /etc/fstab saying something like "if you want to fsck a loopback filesystem use the config file /etc/conf.d/foo" then you would know to look elsewhere yes?

[Nahor <nahor.j+openrc@gmail.com>]

(In reply to comment #11)

Now, if people put a comment in /etc/fstab saying something like "if you want
to fsck a loopback filesystem use the config file /etc/conf.d/foo" then you
would know to look elsewhere yes?

I would read a comment next an already configured loopback.
But I do not know if I would read a comment on how to setup a loopback because I know how such an entry is supposed to be so I would not expect that OpenRC has additional conditions. And if I didn't know how to set one up, I would read howtos on the web and copy/paste/modify their examples.

Other workarounds:

• What about mounting a tmpfs so as to mask the real fstab? i.e. have /etc/fstab be a symlink to /etc/foo/fstab, then during the fsck, mount a tmpfs over /etc/foo? Would that work on the BSDs?
• Or would it be acceptable to use FSTAB_FILE on Linux and other systems when they support it and fallback to your solution when not?

Otherwise your solution is still better than having each loopback user writing their own fsck script. :)

[roy]

(In reply to comment #12)

And if I didn't know how to set one up, I would read

howtos on the web and copy/paste/modify their examples.

All the examples I found don't have it set to fsck at boot :)

Other workarounds:

• What about mounting a tmpfs so as to mask the real fstab? i.e. have

/etc/fstab be a symlink to /etc/foo/fstab, then during the fsck, mount a tmpfs
over /etc/foo? Would that work on the BSDs?

Hmmmm, you would have to overmount /etc and that could copy a lot.
Yes, it would work, but not every OS has tmpfs. For example, we support FreeBSD-6.x, but tmpfs is new in 7 and we couldn't easily do that without tmpfs.

• Or would it be acceptable to use FSTAB_FILE on Linux and other systems when

they support it and fallback to your solution when not?

Yes, but I would not write that script. You of course could ;)

Otherwise your solution is still better than having each loopback user writing
their own fsck script. :)

I'll look into something for this over the next week or so.

[roy]

OK, new proposal - multiplex the fsck script :)

ln -s fsck /etc/init.d/fsck.root
echo 'fsck_passno="=1"' > /etc/conf.d/fsck.root
echo 'rc_need="!fsck fsck.root"' > /etc/conf.d/root
echo 'rc_after="root"' >> /etc/conf.d/fsck

This should work fine with openrc-0.2.2, although the fsck script will spit out the same message twice. I've fixed that in git though.

This makes the following happen
fsck /
mount /
fsck everything_else

And should work with file based mounts on /

Is this OK?

[Nahor <nahor.j+openrc@gmail.com>]

Fine by me. That will fix my particular problem and that's what most distro do anyway.

[roy]

Fixed then :)
Re-open if you have trouble getting it to work.