h2g2 The Hitchhiker's Guide to the Galaxy: Earth Edition

OK, done this before but can;t find the code / remember how I did it.

I have a txt file listing in 1 column a load of unique application file IDs. They are always found in the 2nd line of the datafiles.

What I need to do is, for each number in the file (ie each line therefore) I need to grep the datafile directory for all files that contain that exact number and then rm-f them. Basically we got a shed load of duplicate files and this file id is the only way of telling them from normal ones and therefore deleting them. Sounds simple enough but....
I forgot how to read a file in line by line. I was thinking of using a FOREACH on the file somehow and then using a grep piped to an rm or summats. Without help I will probably end up with some heath robinsonesque loop through the file doing a head -n | tail -1 to obtain the line i want, which seems a bit overkill.

Oh, and before anyone even thinks it, I live in fear and awe of AWK and believe my brain will explode should I ever try to use it!

Have you thought about using Perl for this? This is just the sort of thing Perl is good at. I could probably put together a quick script if you can post an example line from the text file.

File is literally 1 column ie:

1844
1845
1846
17556
17557
17558

only a lot more of them! The other problem is that there are 20,000+ files to search through in the target directory.

Do the filenames follow a pattern, e.g.

somefile1844
somefile1845
somefile1846
somefile17556
somefile17557
somefile17558

or are they all different?

erm, filenames are random numbers, although all starting with '2' helpfully
Basically I want to search everyfile in the directory for each value in turn found in the file of numbers. Any file found to have one of these numbers in I want to remove (eventually, at the moment I just display it).

Currently I have the below but it feels a bit clunky :-


#!/bin/csh
set FILENM = $1
set LINES = 0
set count = 1
set lns = `wc -l $FILENM`
set LINES = `echo $lns | cut -f1 -d' '`
while ( $count <= $LINES )
set INVAL = `head -$count $FILENM | tail -1`
set zinno = `echo $INVAL | cut -f1 -d' '`
echo "*** Files containing"$zinno
foreach ffile ( `find /papps/prod/fds/in_files/dtn_seeb/err_d0019/ -name "2*" -exec grep -l $zinno {} \;` )
echo $ffile
end
@ count ++
end
exit 0

So your filenames are just numbers?

#!/usr/bin/perl

use strict;
use warnings;

# Open the input file (read only)
open(INFILE, "/path/to/inputfile");

# Read in the file line by line
while(<INFILE&gt {
# Remove the trailing \n from the line
chomp;
# If your input file contains naughty characters like spaces
# this will have to be replaced with a regular expression.

my $fullname = "/path/to/files/$_";

# Print or delete
print "$fullname\n";
# unlink $fullname or warn("Unable to remove file $fullname.\n");
}

Whups... I think I've made a bad assumption somewhere along the way...

These files aren't all in one directory, are they? Otherwise there wouldn't be any need to find them... I'll have to do some tweaking to the script I posted. I'll post corrections.

A more sophisticated version that uses "find -name" to locate the files to be deleted:

#!/usr/bin/perl

use strict;
use warnings;

my $inputfilename = shift or die("No source filename was given.\n");

# Open the input file (read only)
open(INFILE, $inputfilename);

# Read in the file line by line
while(<INFILE&gt {
# Remove the trailing \n from the line
chomp;
# If your input file contains naughty characters like spaces
# this will have to be replaced with a regular expression.

# Go to the next line if this one is blank
next unless (/\S+/);

# Use find to get a list of files with the name we just read in
my $foundfilestring = `find -name $_`;

# If there's more than one filename we need to split them
my @foundfiles = split("\n", $foundfilestring);

# Print or delete
print @foundfiles;
# unlink @foundfiles or warn("Unable to remove file $fullname.\n");
}

Grah. Failed to check my work, apparently.

# unlink @foundfiles or warn("Unable to remove file $fullname.\n");

should read:

# unlink @foundfiles;

The variable $fullname was removed from this version so Perl will throw a warning.