Recover Space By Finding Deleted Files That Are Still Held Open.


The other day I was trying to clean out some space on an almost full filesystem that I use to hold some video files. The output from df looked like:

media:~ # df -kh /docs/videos/
Filesystem                  Size  Used Avail Use% Mounted on
/dev/mapper/vgvolume-videos 5.0G  4.2G  526M  90% /docs/videos


I then found the largest file I wanted to delete (a 700M avi video I had recently watched), and removed it. df should now report that I freed up some space right? NOPE!

media:~ # df -kh /docs/videos/
Filesystem                  Size  Used Avail Use% Mounted on
/dev/mapper/vgvolume-videos 5.0G  4.2G  526M  90% /docs/videos


Why wasn’t I able to recover the space on the filesystem? At first I didn’t know. I then decided to unmount and remount the filesystem to see if the changes would take effect. During this process, I found out that I couldn’t unmont the fs because the device was busy:

media:~ # umount /docs/videos/
umount: /docs/videos: device is busy.
    (In some cases useful info about processes that use
     the device is found by lsof(8) or fuser(1))


Ahh, so a program still has some files open on the fs. lsof lets us find out what files.

media:~ # lsof /docs/videos/
COMMAND     PID      USER   FD   TYPE DEVICE  SIZE/OFF NODE NAME
bash      10889 dustymabe  cwd    DIR  253,6      4096    2 /docs/videos
vlc       11244 dustymabe  cwd    DIR  253,6      4096    2 /docs/videos
vlc       11244 dustymabe   11r   REG  253,6 732297098   14 /docs/videos/video1.avi (deleted)
xdg-scree 11281 dustymabe  cwd    DIR  253,6      4096    2 /docs/videos
xprop     11285 dustymabe  cwd    DIR  253,6      4096    2 /docs/videos


So lsof shows us what files are open but it also let me us know something else key to my investigation. The file I had deleted was actually still being held open by VLC media player (I had recently been watching the video and vlc was still up and paused in the middle of playback). lsof let us know that the file had been deleted as well. After closing vlc the space was then released back to the filesystem :)

media:~ # df -kh /docs/videos/
Filesystem                  Size  Used Avail Use% Mounted on
/dev/mapper/vgvolume-videos 5.0G  3.5G  1.2G  75% /docs/videos


As a side note it is worth mentioning that all files that have been deleted but are still be held open by running processes are listed as “(deleted)” when performing an ls -l in the proc file system. For example, to find the file that I deleted that was still being held open by vlc then I would use ls -lR /proc/11244/fd/ | grep deleted. An example of the output is shown below:

media:~ # ls -l /proc/11244/fd/ | grep deleted
lr-x------ 1 dustymabe users 64 Jan 22 21:58 11 -> /docs/videos/video1.avi (deleted)



Cheers!