poline

Python one-liners: Awk-like one-liners for python

Showing:

Popularity

Downloads/wk

0

GitHub Stars

241

Maintenance

Last Commit

4yrs ago

Contributors

3

Package

Dependencies

0

License

UNKNOWN

Categories

Readme

poline

Build Status

poline lets you do awk-like one liners in python.

For example, the following will graph the number of connections for each hosts.

 netstat -an | grep ESTABLISHED | pol "|url(_4).hostname" "counter(_)" ":x, c: Cols(17,40).f(x,'*' * c)"

Example output:

216.58.193.78       *****
74.125.199.189      ****
198.252.206.25      ***
74.125.199.188      **
192.30.253.125      **
127.0.0.1       **
216.58.193.67       *

The equivalent awk version can be found on commandlinefu.

Installation

poline is compatible with Python 2.7 to Python 3.6.

You can easily install poline is via pip:

pip install poline

or

pip<version> install poline

Where <version> is the version of python you wish to use. For example, if you want to use poline with Python 3.6 installed, you should use

pip3.6 install poline

Usage

pol [-h] [-F SEPARATOR] [-s] [-q] expression1 [expression2]

positional arguments:
  expression            python expression

optional arguments:
  -h, --help            show this help message and exit
  -F SEPARATOR, --separator SEPARATOR
                        split each line by SEPARATOR
  -s, --split           split each line
  -q, --quiet           don't implicitly print results

poline stores stdin in the variable _ (underscore) in the form of a generator of strings.

You can see what's inside _ with:

ls -la | pol "x for x in _"
'total 20'
'drwxr-xr-x  2 default root  104 Apr  2 23:34 .'
'drwxr-xr-x 10 default root  230 Apr  2 23:34 ..'
'-rw-r--r--  1 default root    0 Mar 31 14:33 __init__.py'
'-rw-r--r--  1 default root 1740 Apr  2 23:31 _com_collections.py'
'-rw-r--r--  1 default root 4239 Apr  2 23:33 core.py'
'-rw-r--r--  1 default root  750 Apr  2 23:34 fields.py'
'-rw-r--r--  1 default root 3387 Apr  2 23:31 utilfuncs.py'

If you call pol with -s option, you the _ is returned in the form of a generator of lists.

ls -la | pol -s "pformat(x,width=100) for x in _"
['total', '20']
['drwxr-xr-x', '2', 'default', 'root', '104', 'Apr', '2', '23:34', '.']
['drwxr-xr-x', '10', 'default', 'root', '230', 'Apr', '2', '23:34', '..']
['-rw-r--r--', '1', 'default', 'root', '0', 'Mar', '31', '14:33', '__init__.py']
['-rw-r--r--', '1', 'default', 'root', '1740', 'Apr', '2', '23:31', '_com_collections.py']
['-rw-r--r--', '1', 'default', 'root', '4239', 'Apr', '2', '23:33', 'core.py']
['-rw-r--r--', '1', 'default', 'root', '750', 'Apr', '2', '23:34', 'fields.py']
['-rw-r--r--', '1', 'default', 'root', '3387', 'Apr', '2', '23:31', 'utilfuncs.py']

Field class

Each list item is of the class Field, which is derived from String. Field has a number of useful addition methods:

MethodExplanation
i()returns int value of the field
f()returns int value of the field
h()returns field as number of bytes in a human readable format. See bytesize()

Chaining expressions

You can chain expressions as if you were chaining commands on bash using pipes.

netstat -an | grep ESTABLISHED | pol "|url(_4).hostname" "counter(_)" ":x, c: Cols(17,40,None).f(x, get([' '.join(l[1:]) for l in sh(['whois', x],s=T) if 'OrgName' in l[0]], 0), '*' * c)"

Expression types

Awk-like expressions

|epxression

%FS%expression

Works on the last result item by item (or line by line if the last result was stdin) The split() is called |epxression and split(FS) is called with %FS%expression. The list generated by the split is available in __. Individual fields are available as _0, _1, _2 etc.

ps aux | pol "|Cols(10,None).f(_0,_10)"
USER        COMMAND
default     bash
default     ps
default     /tmp/poline/poline_venv/bin/python

lambda-like expressions

:*args:expression Works on the last result item by item (or line by line if the last result was stdin) Receives the n-tuple *args.

Example

ps aux | pol "skip(_)" "|(_0,_10)" ":user,cmd:Cols(50,None).f(next(sh('id',user)),cmd)"
uid=1001(default) gid=0(root) groups=0(root)        bash
uid=1001(default) gid=0(root) groups=0(root)        ps
uid=1001(default) gid=0(root) groups=0(root)        /tmp/poline/poline_venv/bin/python

Utility Functions

We are in the process of adding utility functions to poline. Contributions are most welcome.

barchart(x, p = False, w = 10)

Returns the value x as a barchart. If p is true, x is intepreted as a percentage, with 100% being w wide.

df -B1 | pol "|Cols(20,10,10,10,5,None,10).f(_0,_1.h(),_2.h(),_3.h(),_4, barchart(_2.i()/_1.f(),p=True) if _1.isdigit() else ' '*10,_5)"
Filesystem              1B-blocks   Used        Available   Use%                Mounted
/dev/mapper/docker-8      9.99 G      8.06 G      1.93 G    81%     ▓▓▓▓▓▓▓▓░░  /
tmpfs                    31.37 G      0.00 B     31.37 G    0%      ░░░░░░░░░░  /dev
tmpfs                    31.37 G      0.00 B     31.37 G    0%      ░░░░░░░░░░  /sys/fs/cg
/dev/sda3               211.08 G     26.49 G    173.84 G    14%     ▓░░░░░░░░░  /etc/hosts
shm                      64.00 M      0.00 B     64.00 M    0%      ░░░░░░░░░░  /dev/shm

bytesize(x, u = None, s = False)

Returns the number of bytes x in a human readable string with a 'B', 'K', 'M', 'G', 'T', 'P' prefix.

If x is a string, bytesize tries to convert it to a float, or returns the string as is.

u specifies that bytes are in a different unit, for example u='K' if the size is in 1-K blocks.

s specifies that byte size should be strict.

Example:

$ pol "bytesize(972693249)"
927.63 M

counter(l, n=10)

Sorts a list by descending order, and returns a 2-tuple with element and number of appearances in l:

[(<element>, <count>), ...]

Example:

$ pol -q "pprint(counter('information'), width=20)"
[('i', 2),
 ('n', 2),
 ('o', 2),
 ('f', 1),
 ('r', 1),
 ('m', 1),
 ('a', 1),
 ('t', 1)]

get (l, i, d=None)

get ith element from list l if the ith element exists, or return value d

>>> get([1, 2, 3, 4],1)
2
>>> get([1, 2, 3, 4],4,0)
0

sh (*args, **kwargs)

Executes shell command and arguments specified in args, returns stdout in the form of a list of lists.

Recognized kwargs: F SEPARATOR split each line by SEPARATOR s True|False split each line

Example:

The following displays the inode of each file using stat

Python>=3.6

$ ls | pol "f'{l:20.20}\t%s' % [i[3] for i in sh(['stat',l]) if 'Inode:' in i[2]][0] for l in _"

Python>=2.7

$ ls | pol "[columns(20,None).format(l,i[3]) for i in sh('stat',l,s=True) if 'Inode:' in i[2]][0] for l in _"
LICENSE     360621
Makefile    360653
pol         360606
pol.c       360637
pol.o       360599
README.md   360623

N.B. The syntax of sh has changed, but old syntax is still supported for the sake of backward compatibility.

As well, the popular shell commands cp, df, docker, du, find, git, history, ln, ls, lsof, mv, netstat, nmcli, ps, rm are all functions. For example

They behave as if the command is being passed to sh above.

pol "ls('-lah')"
total 20K
drwxr-xr-x  2 default root  104 Apr  2 23:34 .
drwxr-xr-x 10 default root  230 Apr  2 23:34 ..
-rw-r--r--  1 default root    0 Mar 31 14:33 __init__.py
-rw-r--r--  1 default root 1.7K Apr  2 23:31 _com_collections.py
-rw-r--r--  1 default root 4.2K Apr  2 23:33 core.py
-rw-r--r--  1 default root  750 Apr  2 23:34 fields.py
-rw-r--r--  1 default root 3.4K Apr  2 23:31 utilfuncs.py

skip(iter, n=1)

Skip the first n items from the Iterator iter.

pol "ls('-lah')" "skip(_,n=3)"
-rw-r--r--  1 default root    0 Mar 31 14:33 __init__.py
-rw-r--r--  1 default root 1.7K Apr  2 23:31 _com_collections.py
-rw-r--r--  1 default root 4.2K Apr  2 23:33 core.py
-rw-r--r--  1 default root  750 Apr  2 23:34 fields.py
-rw-r--r--  1 default root 3.4K Apr  2 23:31 utilfuncs.py

Utility classes

Cols

Cols is used to manage formatted output in the form of columns

__init__(*args, **kwargs)

The constructor generators columns by unpacking *args. For example: Cols(10,10,20,None) generates a format string as '{10.10}\t{10.10}\t{20.20}\t{}'

f(*args, **kwargs)

Unpacks *args to pass to String.format(). If format string has not been set, sets the format string as {}\t{}

Examples

The top ten commands you use most often

history | pol "|_1" "counter(_)" ":x,c:Cols(5,None).f(x,c)"

Rate & Review

Great Documentation0
Easy to Use0
Performant0
Highly Customizable0
Bleeding Edge0
Responsive Maintainers0
Poor Documentation0
Hard to Use0
Slow0
Buggy0
Abandoned0
Unwelcoming Community0
100