Data Wrangle

正文

这条命令是在远程服务器执行journalctl， 传回完整日志之后在本地grep，比较慢

ssh myserver journalctl | grep sshd

在远程服务器执行完整命令，在本地less

ssh myserver 'journalctl | grep sshd | grep "Disconnected from"' | less

Save as local file

ssh myserver 'journalctl | grep sshd | grep "Disconnected from"' > ssh.log
less ssh.log

sed 是 stream editor，流编辑器。 sed和grep的区别.....

sed 's/.*Disconnected from //'

意思是：把每行中 Disconnected from 之前的内容删掉，只留下后面的部分。

完整命令

ssh myserver journalctl
 | grep sshd
 | grep "Disconnected from"
 | sed 's/.*Disconnected from //'

命令和参数解释....

带不带g

s/A/B/   每行把第一个 A 换成 B
s/A/B/g  每行把所有 A 换成 B
s/A//    每行删除第一个 A
s/A//g   每行删除所有 A

常见正则表达式

• . means “any single character” except newline
• * zero or more of the preceding match
• + one or more of the preceding match
• [abc] any one character of a, b, and c
• (RX1|RX2) either something that matches RX1 or RX2
• ^ the start of the line
• $ the end of the line

sed’s regular expressions are somewhat weird, and will require you to put a \ before most of these to give them their special meaning. Or you can pass -E.

Which may not be what we wanted. In some regular expression implementations, you can just suffix * or + with a ? to make them non-greedy, but sadly sed doesn’t support that. We could switch to perl’s command-line mode though, which does support that construct:

perl -pe 's/.*?Disconnected from //'

 | sed -E 's/.*Disconnected from (invalid |authenticating )?user .* [^ ]+ port [0-9]+( \[preauth\])?$//'

讲解整个正则表达式

 | sed -E 's/.*Disconnected from (invalid |authenticating )?user (.*) [^ ]+ port [0-9]+( \[preauth\])?$/\2/'

捕获组的使用

Copyssh myserver journalctl \
  | grep sshd \
  | grep "Disconnected from" \
  | sed -E 's/.*Disconnected from (invalid |authenticating )?user (.*) [^ ]+ port [0-9]+( \[preauth\])?$/\2/' \
  | sort \
  | uniq -c

| sort

把用户名排序。 为什么要排序？因为下一步 uniq -c 只能统计相邻的重复行。所以必须先把相同用户名排到一起。 例如：

root
admin
root

排序后：

admin
root
root

uniq 是一个命令，用来合并相邻的重复行。

-c 是 uniq 的参数，表示 count，也就是统计每组重复行出现了几次。 执行：

uniq -c

输出：

2 root3 admin
1 test

注意：uniq 只处理相邻重复行，所以通常要先 sort

ssh myserver journalctl
 | grep sshd
 | grep "Disconnected from"
 | sed -E 's/.*Disconnected from (invalid |authenticating )?user (.*) [^ ]+ port [0-9]+( \[preauth\])?$/\2/'
 | sort | uniq -c
 | sort -nk1,1 | tail -n10

命令，参数以及参数缩写含义解释...

ssh myserver journalctl
 | grep sshd
 | grep "Disconnected from"
 | sed -E 's/.*Disconnected from (invalid |authenticating )?user (.*) [^ ]+ port [0-9]+( \[preauth\])?$/\2/'
 | sort | uniq -c
 | sort -nk1,1 | tail -n10
 | awk '{print $2}' | paste -sd,

命令，参数以及参数缩写含义解释...

awk – another editor

awk is a programming language that just happens to be really good at processing text streams. First, what does {print $2} do? Well, awk programs take the form of an optional pattern plus a block saying what to do if the pattern matches a given line. The default pattern (which we used above) matches all lines. Inside the block, $0 is set to the entire line’s contents, and $1 through $n are set to the nth field of that line, when separated by the awk field separator (whitespace by default, change with -F). In this case, we’re saying that, for every line, print the contents of the second field, which happens to be the username!

Data wrangling to make arguments

数据整理工具配合 xargs 很有用，因为 xargs 可以把前面管道输出的文本变成后面命令的参数。

rustup toolchain list | grep nightly | grep -vE "nightly-x86" | sed 's/-x86.*//' | xargs rustup toolchain uninstall

命令，参数以及参数缩写含义解释...

Wrangling binary data