Display Last Changed Lines in Multiple Files - Python

This is my first English entry. Hereafter, i will write my blog English and Turkish. It depends on

my mood :)

Recently i need to see some product audit logs. However it was saving logs rule and day based folder and file type. There is defined formats for name of files but it is impossible to guess before. Since it was based on ip address or rule n

umber, sth like that. Thus, i needed to check all files that has modified.

getListofFiles: Checks all the files under directory that has been inserted to list or removed by process of writing changed lines. This method prepares filesInfo list which is list of list. Internal list contains one file information. Structure: [[filename, last modified time, last read cursor], [.., ..., ...], ..., ...]

for example:

[['D:\\deneme\\a.txt', 128144

0914.110132, 24L], ['D:\\deneme\\basarili.txt', 1281440962.4076252, 46L], ['D:\\deneme\\t.txt', 1281427352.245864, 32L], ['D:\\deneme\\abc\\1.txt', 1281440315.2405622, 51L], ['D:\\deneme\\abc\\2.log', 1281426866.4594138, 19L]]

changedFiles: Checks last modification

time. If it is different from previous modification time, it calls tailFile methods. TailFile function just reads data after previous cursor to last cursor and writes to outPut File.

Please be careful about command line arguments.

Usage: python multiTail.py directory outputFile

python multiTail D:\\try D:\\out.txt (

outputfile must not be in directory)

Code:

import os
import filecmp
import getopt
import sys

class MultipleTail:
"""This class get the list of text files under a directory which is given by user.
Code writes the changed lines to an output file that is also taken by user.
New file control is made by a while loop. We can think this as an tail -f a directory
Output file rotateion has not implemented yet."""


filesInfos = []
filesDescription = []
directory = ""
outputFile = ""


def __init__(self, inpFile, outFile):
self.filesDescription = []
self.directory = inpFile
self.outputFile = outFile
#self.filesInfos.append(self.filesDescription)

def getListofFiles(self):
for root, dirs, files in os.walk(self.directory):
for fs in files:
notExistFlag = "true"
filepath = root + "\\" + fs
for a in range(len(self.filesInfos)):
if(filepath in self.filesInfos[a]):
notExistFlag = "false"

if(notExistFlag == "true"):
self.filesDescription.append(filepath)
self.filesDescription.append(os.stat(filepath).st_mtime)
self.filesDescription.append(0)
self.filesInfos.append(self.filesDescription)
self.filesDescription = []




def changedFiles(self):
for a in range(len(self.filesInfos)):
if(os.path.exists(self.filesInfos[a][0])):
if((os.stat(self.filesInfos[a][0]).st_mtime != self.filesInfos[a][1]) or (self.filesInfos[a][2] == 0)):
self.filesInfos[a][1] = os.stat(self.filesInfos[a][0]).st_mtime
self.tailFile(self.filesInfos[a][0], self.filesInfos[a][2], a)


def tailFile(self, fileDir, lastCursor, fileOrder):
read_data = ""
with open(fileDir) as tmpInputFile:
tmpInputFile.seek(lastCursor)
endOfFile = os.stat(fileDir).st_size
readByte = endOfFile - lastCursor
read_data = tmpInputFile.read(readByte)
self.filesInfos[fileOrder][2] = endOfFile

outFile = open(self.outputFile, "a+")
print read_data
outFile.write(read_data)
tmpInputFile.close()
outFile.close()


if __name__ == "__main__":
try:
opts, args = getopt.getopt(sys.argv[1:], "ho:v", ["help", "output="])
except getopt.GetoptError, err:
print str(err)
usage()
sys.exit(2)
output = None
verbose = False
for o, a in opts:
if o == "-v":
verbose = True
elif o in ("-h", "--help"):
usage()
sys.exit()
elif o in ("-o", "--output"):
output = a
else:
assert False, "unhandled option"

inFile = sys.argv[1]
oFile = sys.argv[2]

ObjectTail = MultipleTail(inFile, oFile)
while 1:
print ObjectTail.filesInfos
ObjectTail.getListofFiles()
ObjectTail.changedFiles()

Result: