At 4:30, I had this script running, and at 4:45 the data was scrubbed. Now granted, it's a quick hack, but that is what python is good for. I was even able to put a few flags to just get a count of records etc... But this little 30 minute job probably saved weeks of head banging. It simply replaces funny characters with an underscore.
I work on so many large projects that can take more than a year to complete that I love the occasional small job. Kind of like painting a room, you can leave work at the end of the day and feel like you really did something.
Code: Select all
#################################################
# python scrub.py -i <infile> -o<outfile> -v -c
#
# -c just returns a count of errors with no processing
# -v verbose.
#################################################
import getopt
import sys
def fix(line):
fixed_line = ''
for c in line:
if ord(c) > 127:
c = '_'
fixed_line = fixed_line + c
return fixed_line
verbose_flag = 0
justcount_flag = 0
infile = '/tmp/input.file'
outfile = '/tmp/output.file'
try:
optlist,args = getopt.getopt(sys.argv[1:],"i:o:vc")
except getopt.error, error_msg:
print error_msg
print 'Usage: scrub.py -i<infile> -o<outfile> -v'
sys.exit(1)
for opt in optlist:
if opt[0] == '-i':
infile = opt[1]
elif opt[0] == '-o':
outfile = opt[1]
elif opt[0] == '-v':
verbose_flag = 1
elif opt[0] == '-c':
try:
infd = open(infile)
except:
print 'Error opening input file ' + infile
sys.exit(1)
try:
outfd = open(outfile,'w')
except:
print 'Error opening output file ' + outfile
sys.exit(1)
ctr = 0
for line in infd.readlines():
try:
unicode(line,'utf-8')
outfd.write(line)
except:
if justcount_flag == 0:
outfd.write(fix(line))
ctr = ctr + 1
print "Found " + str(ctr) + " errors"
