The other day I was trying to clean out some space on an almost full
filesystem that I use to hold some video files. The output from df
looked like:
\
media:~ # df -kh /docs/videos/
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vgvolume-videos 5.0G 4.2G 526M 90% /docs/videos
I then found the largest file I wanted to delete (a 700M avi video I had
recently watched), and removed it. df should now report that I freed
up some space right? NOPE!
\
media:~ # df -kh /docs/videos/
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vgvolume-videos 5.0G 4.2G 526M 90% /docs/videos
Why wasn’t I able to recover the space on the filesystem? At first I
didn’t know. I then decided to unmount and remount the filesystem to see
if the changes would take effect. During this process, I found out that
I couldn’t unmont the fs because the device was busy:
\
media:~ # umount /docs/videos/
umount: /docs/videos: device is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))
Ahh, so a program still has some files open on the fs. lsof lets us
find out what files.
\
media:~ # lsof /docs/videos/
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
bash 10889 dustymabe cwd DIR 253,6 4096 2 /docs/videos
vlc 11244 dustymabe cwd DIR 253,6 4096 2 /docs/videos
vlc 11244 dustymabe 11r REG 253,6 732297098 14 /docs/videos/video1.avi (deleted)
xdg-scree 11281 dustymabe cwd DIR 253,6 4096 2 /docs/videos
xprop 11285 dustymabe cwd DIR 253,6 4096 2 /docs/videos
So lsof shows us what files are open but it also let me us know
something else key to my investigation. The file I had deleted was
actually still being held open by VLC media player (I had recently been
watching the video and vlc was still up and paused in the middle of
playback). lsof let us know that the file had been deleted as well.
After closing vlc the space was then released back to the filesystem :)\
media:~ # df -kh /docs/videos/
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vgvolume-videos 5.0G 3.5G 1.2G 75% /docs/videos
As a side note it is worth mentioning that all files that have been
deleted but are still be held open by running processes are listed as
“(deleted)” when performing an ls -l in the proc file system. For
example, to find the file that I deleted that was still being held open
by vlc then I would use ls -lR /proc/11244/fd/ | grep deleted. An
example of the output is shown below:
\
media:~ # ls -l /proc/11244/fd/ | grep deleted
lr-x------ 1 dustymabe users 64 Jan 22 21:58 11 -> /docs/videos/video1.avi (deleted)
Cheers!