Hug Me (hugs) wrote in freebsd_users,
Hug Me

Delete a million files?

One of the devices I admin is an old Solaris 8 device... I was cleaning it up yesterday and found a directory of spam from 2002... good candidate for deletion... I did an `rm -Rf` on the directory and left it alone. Today I go and check to see if it's complete... nope, only 2% done, it had been running for 14 hours. This is odd...

A closer inspection of the directory and I found that there were in fact over 2,000,000 emails in this directory, each in their own directory and file. The system was taking about half a second PER FILE. There has to be a better way.

Some research online I got no result. There are a lot of people with this problem but there weren't any answers which worked. People had tried to script something in perl, php, shell... no result.

So I got to thinking about it, the `rm` command is single threaded... doesn't really use up that much cpu or disk.. so you can run MANY of them at once with out hurting the device (this is a production device btw). BUT if you try to run a rm command for EVERY file in the directory, THAT will hurt the device. What about running a number of remove commands at once, when one stops the next one can start. Sounds like a plan.

It works! here is the command I wrote... in this case the number 5 controls the number of commands you want it to run at the same time... right now I have it set for 50 and I'm deleting 60,000 files about every 2 minutes.

while read X; do while [ "$(jobs | wc -l)" -gt 5 ]; do sleep 1; done; rm -R $X & done< <(ls) 
  • Post a new comment


    default userpic
    When you submit the form an invisible reCAPTCHA check will be performed.
    You must follow the Privacy Policy and Google Terms of use.