Core utilities/Tips and tricks - ArchWiki

# Core utilities/Tips and tricks

See Core utilities for the main article.

## grep

### Colored output

`grep`'s color output can be helpful for learning regexp and additional `grep` functionality.

To enable grep coloring write the following entry to the shell configuration file (e.g. if using Bash):

    ~/.bashrc

    alias grep='grep --color=auto'

To include file line numbers in the output, add the option `-n` to the line.

The environment variable `GREP_COLOR` can be used to define the default highlight color (the default is red). To change the color find the ANSI escape sequence for the color liked and add it:

    export GREP_COLOR="1;32"

`GREP_COLORS` may be used to define specific searches.

## less

### Colored output through environment variables

Add the following lines to your shell configuration file:

    ~/.bashrc

    export LESS=-R
    export LESS_TERMCAP_mb=$'\E[1;31m'
    export LESS_TERMCAP_md=$'\E[1;36m'
    export LESS_TERMCAP_me=$'\E[0m'
    export LESS_TERMCAP_se=$'\E[0m'
    export LESS_TERMCAP_so=$'\E[01;44;33m'
    export LESS_TERMCAP_ue=$'\E[0m'
    export LESS_TERMCAP_us=$'\E[1;32m'

Change the values (ANSI escape code) as you like.

Note: The `LESS_TERMCAL_xx` variables is currently undocumented in less(1), for a detailed explanation on these sequences, see this answer.

### Colored output through wrappers

You can enable code syntax coloring in less. First, install source-highlight, then add these lines to your shell configuration file:

    ~/.bashrc

    export LESSOPEN="| /usr/bin/source-highlight-esc.sh %s"
    export LESS='-R '

Frequent users of the command line interface might want to install lesspipe.

Users may now list the compressed files inside of an archive using their pager:

    $ less compressed_file.tar.gz

    ==> use tar_file:contained_file to view a file in the archive
    -rw------- username/group  695 2008-01-04 19:24 compressed_file/content1
    -rw------- username/group   43 2007-11-07 11:17 compressed_file/content2
    compressed_file.tar.gz (END)

lesspipe also grants less the ability of interfacing with files other than archives, serving as an alternative for the specific command associated for that file-type (such as viewing HTML via python-html2text).

Re-login after installing lesspipe in order to activate it, or source `/etc/profile.d/lesspipe.sh`.

### Vim as alternative pager

Vim (visual editor improved) has a script to view the content of text files, compressed files, binaries, directories. Add the following line to your shell configuration file to use it as a pager:

    ~/.bashrc

    alias less='/usr/share/vim/vim74/macros/less.sh'

There is also an alternative to less.sh macro, which may work as the `PAGER` environment variable. Install vimpager and add the following to your shell configuration file:

    ~/.bashrc

    export PAGER='vimpager'
    alias less=$PAGER

Now programs that use the `PAGER` environment variable, like git, will use vim as pager.

### Colored output when reading from stdin

Note: It is recommended to add #Colored output through environment variables to your `~/.bashrc` or `~/.zshrc`, as the below is based on `export LESS=R`

When you run a command and pipe its standard output (stdout) to less for a paged view (e.g. `pacman -Qe | less`), you may find that the output is no longer colored. This is usually because the program tries to detect if its stdout is an interactive terminal, in which case it prints colored text, and otherwise prints uncolored text. This is good behaviour when you want to redirect stdout to a file, e.g. `pacman -Qe > pkglst-backup.txt`, but less suited when you want to view output in `less`.

Some programs provide an option to disable the interactive tty detection:

    # dmesg --color=always | less

In case that the program does not provide any similar option, it is possible to trick the program into thinking its stdout is an interactive terminal with the following utilities:

- stdoutisatty — A small program which catches the `isatty` function call.

  https://github.com/lilydjwg/stdoutisatty. || stdoutisatty-gitAUR
  Example: `stdoutisatty program | less`

- unbuffer — A tclsh script comes with expect, it invokes desired program within a pty.

  http://expect.sourceforge.net/example/unbuffer.man.html || expect
  Example: `unbuffer program | less`

Alternatively, using zpty module from zsh: [1]

    ~/.zshrc

    zmodload zsh/zpty

    pty() {
    	zpty pty-${UID} ${1+$@}
    	if [[ ! -t 1 ]];then
    		setopt local_traps
    		trap '' INT
    	fi
    	zpty -r pty-${UID}
    	zpty -d pty-${UID}
    }

    ptyless() {
    	pty $@ | less
    }

Usage:

    $ ptyless program

To pipe it to other pager (less in this example):

    $ pty program | less

## ls

### Colored output

Colored output can be enabled with a simple alias. File `~/.bashrc` should already have the following entry copied from `/etc/skel/.bashrc`:

    alias ls='ls --color=auto'

The next step will further enhance the colored ls output; for example, broken (orphan) symlinks will start showing in a red hue. Add the following to your shell configuration file:

    eval $(dircolors -b)

## tar

### Backup with parallel compression

This article or section is a candidate for merging with Backup.

Notes: This may be better suited for a dedicated backup article (Discuss in Talk:Core utilities/Tips and tricks#)

To back up using parallel compression (SMP), use pbzip2 (Parallel bzip2).

First back up the files to a plain tarball with no compression:

    # tar -cvf /destionation_path/etc-backup.tar /etc

Then use pbzip2 to compress it in parallel:

    $ pbzip2 /path/to/chosen/directory/etc-backup.tar.bz2

Store `etc-backup.tar.bz2` on one or more offline media, such as a USB stick, external hard drive, or CD-R. Occasionally verify the integrity of the backup process by comparing original files and directories with their backups. Possibly maintain a list of hashes of the backed up files to make the comparison quicker.

Restore corrupted `/etc` files by extracting the `etc-backup.tar.bz2` file in a temporary working directory, and copying over individual files and directories as needed. To restore the entire `/etc` directory with all its contents execute the following command as root:

    tar -xvjf etc-backup.tar.bz2 -C /

### Restore name of a tar.gz archive

This section is being considered for removal.

Reason: Those scripts have a code issue, and shouldn't be mantained here anyway (Discuss in Talk:Core utilities/Tips and tricks#)

Here describes about how you can restore normal names for the most of a recovered tar.gz archives by photorec. First you need to create a file with a more detailed information about files, see: post recovery tasks. Restored tar.gz files with photorec might look like `./recup_dir.996/f864593944_wmmaiload-1.0.5.tar.gz` or `./recup_dir.996/f864589184.tar.gz`.

The tar.gz archive includes inside a tar archive whose name is used to restore filename of the tar.gz if it is possible or name of a first folder/file from inside of the archive.

The ways that might be used to restore the name of the tar.gz archives:

- From part of a name by cutting everything before '_' in files with created by photorec with pattern: f864593944_wmmaiload-1.0.5.tar.gz
- From name of a tar inside of the tar.gz archive.
- From the name of the folder inside it. In many cases it usually compressed a whole folder that might be similar to the archive name.

Note: This script restores only by separating original name from the photorec generated name or use a name of the first folder or file inside of the archive.
To restore all of the files that contain only the original name as part of the generated you can download the restore-orignames-only.sh script from the SourceForge website, but it will add the Duplicate123 (123 is number of a processed file) to files with the same name in the destination directory.

#### Collect info data about tar.gz

Collecting needed data from inside of the archives in order to restore their names.

    collect-info-about-tar-gz.sh

    #!/bin/bash

    FileName="$1"
    IntCheck=$(7z t "$1" | grep ^"Everything is Ok"$ )
    AddIt="False"

    if [ 'XX'"$IntCheck" != 'XXEverything is Ok'   ]; then ErrorFlag="Damaged";else ErrorFlag='OK';fi;

    if [ 'XX'"$(echo "$FileName" | grep -i tar.gz$ )" != 'XX'  ];then
    AddIt='True'

    BaseFileName=$(tar -tvf "$FileName" | head -1 | awk '{ print substr($0, index($0,$6)) }' )
    OutOfOrig="$( echo "$1" | awk '{ print substr($0, index($0,$4)) }' )"
    if [ 'XX'"$(echo "${BaseFileName}" | grep '/')" == 'XX' ]; then

    BaseFileName="$(echo "${BaseFileName}"|  sed 's/\///m')"
    fi
    fi

    if [ $AddIt == 'True'  ]; then

    if [ 'XX'"${OutOfOrig}" != 'XX'   ]; then IfOrig=$(echo "$OutOfOrig" | grep -v '/') ;fi
    if [ 'XX'"${IfOrig}" != 'XX'   ]; then BaseFileName="$IfOrig" ; fi
    BF="$(echo "$BaseFileName" | sed 's/\///g' | sed 's/^\.//m' )"
    echo "$1"'|'"${BF}"'.tar.gz''|'Errors:'|'$ErrorFlag'|' >> /tmp/tmpDB.txt
    fi;

Run `find -type f -name *.tar.gz -exec collect-info-about-tar-gz.sh "{}"\;`

#### Restore filenames

This will restore file names based on collected ínfo about files.

    restore-tar-gz-names.sh

    #!/bin/bash
    CountAll=0

    Destination="$HOME/tar-gz"
    mkdir -v "$(echo $Destination)" -p

      ArrayFillCount=0;
    while read line ; do
      ArrayFillCount=$((ArrayFillCount+1))
          ArrayOfFiles[$ArrayFillCount]=$line
    done < /tmp/tmpDB.txt;

    XX=${#ArrayOfFiles[@]}
    echo $XX
    while [  $XX != $CountAll ] ; do
    FolderName="$(echo ${ArrayOfFiles[$CountAll]} | awk -F'|' '{print $2}' | cut -c1-4 | tr '[:lower:]'  '[:upper:]' )"
    PathToFile="$(echo ${ArrayOfFiles[$CountAll]} | awk -F'|' '{print $1}')"
      FileName="$(echo ${ArrayOfFiles[$CountAll]} | awk -F'|' '{print $2}')"
    #SourceFolder="$(echo $PathToFile | sed s/$(basename $PathToFile)//m )"
     SourceFolder='./Sorted/'

    if [  "$(echo $FolderName)" ==  '.TAR'  ];then
       FileName="$(echo ${ArrayOfFiles[$CountAll]} | awk -F'_' '{ print substr($0, index($0,$4))}'|awk -F'|' '{print $1}' )"
       FolderName="$(echo $FileName | cut -c1-4 | tr '[:lower:]'  '[:upper:]' )"
    fi
    if [ -f "$FileName" ]; then
         FIX="$(echo $PathToFile | sed s/$(basename $PathToFile)//m )"
         BadName="$(echo $PathToFile | awk -F"$FIX" '{print $2}' )"
         if [ -d "${Destination}/BadName"  ];then echo A > /dev/null;else mkdir "${Destination}/BadName";fi
         cp -fv "${PathToFile}" "${Destination}"'/'BadName'/'"$BadName"
    else
      IfExist="$(echo $Destination/${SourceFolder}${FolderName})"
      if [ -d "$IfExist"  ]; then echo a > /dev/null;else mkdir -vp "$(echo $Destination/${SourceFolder}${FolderName})"; fi
         if [ -f "$Destination/${SourceFolder}${FolderName}"'/'"$FileName"   ];
         then DupName="${FileName}"_Duplicate"${CountAll}";
    IfExist="$(echo $Destination/Duples)"
    if [ -d "$IfExist"  ]; then echo a > /dev/null;
    cp -fv "${PathToFile}" "$Destination/Duples"'/'"$DupName";
    else
    mkdir -vp "$(echo $Destination/Duples)";
    cp -fv "${PathToFile}" "$Destination/Duples"'/'"$DupName";fi
      else
    cp -fv "${PathToFile}" "$Destination/${SourceFolder}${FolderName}"'/'"$FileName"
      fi

    fi
    CountAll=$((CountAll+1))
    echo $CountAll
    done;
    echo $XX

Files will be restored with pattern like `$HOME/tar-gz/Sorted/MIXE` where MIXE is the first 4 letters of a filename. You can adjust it with the `cut -c1-4` command inside the script. In the `$HOME/tar-gz/BadName` places damaged files or files where it was impossible to get filename. In the `$HOME/tar-gz/Duples` places duplicates of files with pattern filename.tar.gz_Duplicate123 where 123 is a number of a processed file.

Categories:

- System administration
- Command shells

Hidden categories:

- Pages or sections flagged with Template:Merge
- Sections flagged with Template:Remove