代码语言
.
CSharp
.
JS
Java
Asp.Net
C
MSSQL
PHP
Css
PLSQL
Python
Shell
EBS
ASP
Perl
ObjC
VB.Net
VBS
MYSQL
GO
Delphi
AS
DB2
Domino
Rails
ActionScript
Scala
代码分类
文件
系统
字符串
数据库
网络相关
图形/GUI
多媒体
算法
游戏
Jquery
Extjs
Android
HTML5
菜单
网页交互
WinForm
控件
企业应用
安全与加密
脚本/批处理
开放平台
其它
【
Python
】
python实现日志的查询、统计
作者:
水熊宝宝
/ 发布于
2013/8/5
/
854
简单的日志统计是不需要使用重量级的Hadoop,我用python实现了日志的统计。原理是用fabric登录到远程linux,组合使用grep、uniq、sort、awk对日志进行操作,可以根据正则表达式指定规则抽取符合规则的日志,做查询,计数,分类统计。 注意:要安装fabric库
#encoding=utf-8 from fabric.api import run,env,local,cd from fabric.tasks import execute,abort from fabric.contrib.console import confirm import logging logging.basicConfig(format='[%(levelname)s]: %(message)s', level=logging.DEBUG) logger = logging.getLogger(__name__) logging.getLogger('paramiko.transport').setLevel(logging.ERROR) logger.setLevel(logging.DEBUG) EXECUTE_RESULT = {} def hosts(hostarr): ''' set hosts hostarr:[(hostname,password),(hostname,password)...] ''' env.hosts = [x[0] for x in hostarr] env.passwords = dict(x for x in hostarr) def query(expression,hostname,logfile,unique=True,sort=None,output=None,pattern=None,path=None): ''' expression: regex rule hostname: hostname as specified hosts() logfile: log file name, wildcard supported, eg:*.log unique: whether result is unique sort: 1(ASC) or -1(DESC) ,default None output:None or file name, default None imply print stream pattern: group pattern , default None imply '1' path: cd to path before execution ''' if not path: path = r'.' cmd_str = generate_cmd(expression,logfile,unique,sort,output,pattern) execute(executor,hostname,cmd_str,path,host=hostname) result = EXECUTE_RESULT[hostname] return result def aggregate(expression,hostname,logfile,output=None,pattern=None,path=None): ''' expression: regex rule hostname: hostname as specified hosts() logfile: log file name, wildcard supported, eg:*.log output:None or file name, default None imply print stream pattern: group pattern , default None imply '1' path: cd to path before execution ''' if not path: path = r'.' cmd_str = generate_cmd(expression,logfile,False,None,output,pattern,True,True) execute(executor,hostname,cmd_str,path,host=hostname) result = EXECUTE_RESULT[hostname] return result def count(expression,hostname,logfile,unique=True,sort=None,output=None,pattern=None,path=None): ''' expression: regex rule hostname: hostname as specified hosts() logfile: log file name, wildcard supported, eg:*.log unique: whether result is unique sort: 1(ASC) or -1(DESC) ,default None output:None or file name, default None imply print stream pattern: group pattern , default None imply '1' path: cd to path before execution ''' if not path: path = r'.' cmd_str = generate_cmd(expression,logfile,unique,sort,output,pattern,True) execute(executor,hostname,cmd_str,path,host=hostname) result = EXECUTE_RESULT[hostname] if result: result = int(result[0]) return result def executor(hostname,cmd_str,path=None): ''' executor , called by execute ''' if not path: path = r'.' with cd(path): res = run(cmd_str,quiet=True) logger.debug('Command: %s:%s > %s'%(hostname,path,cmd_str)) logger.debug('Command Execute Successful:%s, Failure:%s'%(res.succeeded,res.failed)) EXECUTE_RESULT[hostname] = res.splitlines() def generate_cmd(expression,logfile,unique=True,sort=None,output=None,pattern=None,count=False,aggregate=False): ''' generate command ''' if not pattern: pattern = r'\1' if aggregate: aggregate = '''| awk '{a[$1]++}END{for (j in a) print j","a[j]}' ''' unique = False sort = False count = False else: aggregate = '' if not unique: unique = '' else: unique = '| uniq' if sort: if sort==1: sort = '| sort' elif sort==-1: sort = '| sort -r' else: sort = '' else: sort = '' if count: count = '| wc -l' else: count = '' if output: output = '>%s'%output else: output = '' cmd_str = '''cat %s | grep "%s" | sed 's/%s/%s/gp' %s %s %s %s %s'''%(logfile,expression,expression,pattern,unique,sort,count,output,aggregate) return cmd_str
试试其它关键字
日志的查询
同语言下
.
比较两个图片的相似度
.
过urllib2获取带有中文参数的url内容
.
不下载获取远程图片的宽度和高度及文件大小
.
通过qrcode库生成二维码
.
通过httplib发送GET和POST请求
.
Django下解决小文件下载
.
遍历windows的所有窗口并输出窗口标题
.
根据窗口标题调用窗口
.
python 抓取搜狗指定公众号
.
pandas读取指定列
可能有用的
.
C#实现的html内容截取
.
List 切割成几份 工具类
.
SQL查询 多列合并成一行用逗号隔开
.
一行一行读取txt的内容
.
C#动态修改文件夹名称(FSO实现,不移动文件)
.
c# 移动文件或文件夹
.
c#图片添加水印
.
Java PDF转换成图片并输出给前台展示
.
网站后台修改图片尺寸代码
.
处理大图片在缩略图时的展示
水熊宝宝
贡献的其它代码
(
2
)
.
python实现日志的查询、统计
.
监测tomcat是否宕机,控制重启
Copyright © 2004 - 2024 dezai.cn. All Rights Reserved
站长博客
粤ICP备13059550号-3