🔐
SecWiki
  • Home
  • General
    • Interesting Links
      • Curriculum
    • Pentest Labs, Wargames Sites
      • How To Vulnhub with VirtualBox
  • Network Pentest
    • Courses
      • TCM - Zero to Hero
        • Week 1: Setup
          • ipsweep.sh
        • Week 2: Python 101
          • python101.py
          • bof.py
        • Week 3: Python 102
          • python102.py
          • scanner.py
        • Week 4: Passive OSINT
        • Week 5: Scanning Tools & Tactics
          • nmap
          • Nessus
          • msfconsole
        • Week 6: Enumeration
        • Week 7: Exploitation, Shells, and Some Credential Stuffing
        • Week 8: LLMNR/NBT-NS Poisoning
        • Week 9: NTLM
        • Week 10: MS17-010, GPP/cPasswords, and Kerberoasting
        • Week 11: File Transfers, Pivoting, Reporting
        • Commands
      • Penetration Testing Student (PTS)
      • OSCP Study
    • Recon
      • OSINT
    • Enumeration
      • Samba Shares
      • ProFtpd
    • Gaining Access
      • Reverse Shells
    • Privilege Escalation
      • Meterpreter
      • Spawning a TTY Shell
      • Reverse Shell Cheat Sheet
      • Cracking Hashes
      • Restricted Linux Shell Escape
      • Linux Privilege Escalation
        • lxd
        • sytemctl
      • Windows Privilege Escalation
        • Active Directory
          • What is AD?
        • User Enumeration
    • Post Exploitation
      • Cleanup
      • Maintaining Access
      • Pivoting
      • File Transfers
      • Covering Tracks
    • Vulnerabilities Checklist
    • Report Writing
  • Web App Pentest
    • Tools
      • Burp Suite
      • THC-Hydra BruteForce
    • Injection
      • SQL Injection
    • Broken Authentication
    • Sensitive Data Exposure
      • SQLite3
    • XML External Entity
      • XML Background
      • XPath Injection
    • Broken Access Control
    • Security Misconfiguration
    • Upload/Download
      • Download Bypass: Poison Null Byte
    • XSS
      • DOMXSS
      • Persistent XSS
      • Reflected (Client-side) XSS
      • Data URLs
    • Insecure Deserialization
    • Components with Known Vulnerabilities
    • Insufficient Logging and Monitoring
    • Server-Side Request Forgery (SSRF)
  • CTF
    • Intro to CTF
    • Forensics
      • Challenges
    • Steganography
    • Reverse Engineering
    • Tools
  • Network Security
    • Courses
      • Sec+
      • IBM Cybersecurity Analyst Professional Certificate
      • ISCI CNSS Course
        • Introduction to Network Security
          • Network Basics
          • Basic Network Utilities
          • The OSI Model
          • Threat Classification
          • Security Terminology
          • Approaches of Network Security
          • Law and Network Security
        • Types of Attacks
          • Denial of Service Attacks
          • Buffer Overflow Attacks
          • IP Spoofing
          • Session Hijacking
        • Fundamentals of Firewalls
          • What is a Firewall
          • Firewall Types
          • Firewall Implementation
          • Proxy Servers
          • Windows Firewalls
          • Linux Firewalls
        • Intrusion-Detection Systems
          • IDS Concepts
          • Components and Processes of IDS
          • Implementing IDS
          • Honeypots
        • Fundamentals of Encryption
          • The History of Encryption
          • Modern Encryption Methods
          • Windows and Linux Encryption
          • Hashing
          • Cracking Passwords
        • Virtual Private Networks (VPN)
          • Introduction to VPN
          • VPN Protocols
          • IPSec
          • SSL/TLS
          • VPN Solutions
        • Operating System Hardening
          • Configuring Windows
          • Configuring Linux
          • Operating System Patches
        • Virus Attacks and How to Defend
          • Virus Types and Attacks
          • Virus Scanners
          • Antivirus
          • Virus Infection and Identification
          • Trojan Horses
          • Spyware or Adware
        • Security Policies
          • User Policies Definition
          • System Administration Policies
          • Access Control
        • Assessing System Security
          • Risk Assessment
          • Conducting an Initial Assessment
          • Probing the Network
          • Vulnerabilities
          • Documenting Security
        • Security Standards
          • ISO Standards
          • NIST Standards
          • General Data Protection Regulation (GDPR)
          • PCI DSS
        • Physical Security and Recovery
          • Physical Security
          • Disaster Recovery
          • Fault Tolerance
        • Attackers Techniques
          • Hacking Preparation
          • The Attack Phase
          • Hacking Wi-Fi
    • The Web
    • The OSI Model
    • Malware Traffic Analysis with Wireshark
  • Digital Forensics
    • Autopsy - open-source digital forensics platform
  • Exploit Dev/Analysis
    • Code Review
      • Tools
    • Buffer Overflows
    • Static Analysis
      • Antivirus Scanning
      • Hashing
      • File strings
      • Packed and Obfuscated Malware
        • Demo: UPX
      • Portable Executable File Format (PE)
        • Tools
        • Linked Libraries and Functions
        • PE File Headers and Sections
  • Shell
    • ./missing-semester
      • Course overview + the shell
      • Shell Tools and Scripting
      • Editors (Vim)
      • Data Wrangling
      • Command-line Environment
    • Bash Tricks
    • .bashrc
    • Random Commands
      • sed
  • Hardware
    • NAND2Tetris
      • Boolean Functions and Gate Logic
      • Boolean Arithmetic and the ALU
      • Memory
      • Machine Language
      • Computer Architecture
      • Assembler
  • Other
    • K8s
      • Chapter 1: From Monolith to Microservices
      • Chapter 2: Container Orchestration
      • Chapter 3: Kubernetes
      • Chapter 4: Kubernetes Architecture
Powered by GitBook
On this page
  • Shell Scripting
  • Bash Basics
  • Bash Special Variables
  • Return Codes
  • Command and Process Substitution
  • Globbing
  • Shellcheck
  • Shebangs
  • Shell Functions vs Scripts
  • Shell Tools
  • Finding how to use commands
  • Finding files
  • Finding code
  • Finding shell commands
  • Directory Navigation
  • Exercises

Was this helpful?

  1. Shell
  2. ./missing-semester

Shell Tools and Scripting

Shell Scripting

Bash Basics

To assign variables in bash, use the syntax foo=bar and access the value of the variable with $foo. Note that foo = bar will not work since it is interpreted as calling the foo program with arguments = and bar. In general, in shell scripts the space character will perform argument splitting. This behavior can be confusing to use at first, so always check for that.

Strings in bash can be defined with ' and " delimiters, but they are not equivalent. Strings delimited with ' are literal strings and will not substitute variable values whereas " delimited strings will.

foo=bar
echo "$foo"
# prints bar
echo '$foo'
# prints $foo

As with most programming languages, bash supports control flow techniques including if, case, while and for. Similarly, bash has functions that take arguments and can operate with them. Here is an example of a function that creates a directory and cds into it.

mcd () {
    mkdir -p "$1"
    cd "$1"
}

Here $1 is the first argument to the script/function.

Bash Special Variables

  • $0 - Name of the script

  • $1 to $9 - Arguments to the script. $1 is the first argument and so on.

  • $@ - All the arguments

  • $# - Number of arguments

  • $? - Return code of the previous command

  • $$ - Process identification number (PID) for the current script

  • !! - Entire last command, including arguments. A common pattern is to execute a command only for it to fail due to missing permissions; you can quickly re-execute the command with sudo by doing sudo !!

  • $_ - Last argument from the last command. If you are in an interactive shell, you can also quickly get this value by typing Esc followed by .

Return Codes

The return code or exit status is the way scripts/commands have to communicate how execution went. A value of 0 usually means everything went OK; anything different from 0 means an error occurred.

false || echo "Oops, fail"
# Oops, fail

true || echo "Will not be printed"
#

true && echo "Things went well"
# Things went well

false && echo "Will not be printed"
#

true ; echo "This will always run"
# This will always run

false ; echo "This will always run"
# This will always run

Command and Process Substitution

Another common pattern is wanting to get the output of a command as a variable. This can be done with command substitution. Whenever you place $( CMD ) it will execute CMD, get the output of the command and substitute it in place. For example, if you do for file in $(ls), the shell will first call ls and then iterate over those values. A lesser known similar feature is process substitution, <( CMD ) will execute CMD and place the output in a temporary file and substitute the <() with that file’s name. This is useful when commands expect values to be passed by file instead of by STDIN. For example, diff <(ls foo) <(ls bar) will show differences between files in dirs foo and bar.

#!/bin/bash

echo "Starting program at $(date)" # Date will be substituted

echo "Running program $0 with $# arguments with pid $$"

for file in "$@"; do
    grep foobar "$file" > /dev/null 2> /dev/null
    # When pattern is not found, grep has exit status 1
    # We redirect STDOUT and STDERR to a null register since we do not care about them
    if [[ $? -ne 0 ]]; then
        echo "File $file does not have any foobar, adding one"
        echo "# foobar" >> "$file"
    fi
done

Globbing

  • Wildcards - Whenever you want to perform some sort of wildcard matching, you can use ? and * to match one or any amount of characters respectively. For instance, given files foo, foo1, foo2, foo10 and bar, the command rm foo? will delete foo1 and foo2 whereas rm foo* will delete all but bar.

  • Curly braces {} - Whenever you have a common substring in a series of commands, you can use curly braces for bash to expand this automatically. This comes in very handy when moving or converting files.

convert image.{png,jpg}
# Will expand to
convert image.png image.jpg

cp /path/to/project/{foo,bar,baz}.sh /newpath
# Will expand to
cp /path/to/project/foo.sh /path/to/project/bar.sh /path/to/project/baz.sh /newpath

# Globbing techniques can also be combined
mv *{.py,.sh} folder
# Will move all *.py and *.sh files


mkdir foo bar
# This creates files foo/a, foo/b, ... foo/h, bar/a, bar/b, ... bar/h
touch {foo,bar}/{a..h}
touch foo/x bar/y
# Show differences between files in foo and bar
diff <(ls foo) <(ls bar)
# Outputs
# < x
# ---
# > y

Shellcheck

Shebangs

Note that scripts need not necessarily be written in bash to be called from the terminal. For instance, here’s a simple Python script that outputs its arguments in reversed order:

#!/usr/local/bin/python
import sys
for arg in reversed(sys.argv[1:]):
    print(arg)

Shell Functions vs Scripts

  • Functions have to be in the same language as the shell, while scripts can be written in any language. This is why including a shebang for scripts is important.

  • Functions are loaded once when their definition is read. Scripts are loaded every time they are executed. This makes functions slightly faster to load, but whenever you change them you will have to reload their definition.

  • As with any programming language, functions are a powerful construct to achieve modularity, code reuse, and clarity of shell code. Often shell scripts will include their own function definitions.

Shell Tools

Finding how to use commands

man, -h, --help,:help,?, TLDR pages: tldr

Finding files

# Find all directories named src
find . -name src -type d
# Find all python files that have a folder named test in their path
find . -path '*/test/*.py' -type f
# Find all files modified in the last day
find . -mtime -1
# Find all zip files with size in range 500k to 10M
find . -size +500k -size -10M -name '*.tar.gz'

Beyond listing files, find can also perform actions over files that match your query. This property can be incredibly helpful to simplify what could be fairly monotonous tasks.

# Delete all files with .tmp extension
find . -name '*.tmp' -exec rm {} \;
# Find all PNG files and convert them to JPG
find . -name '*.png' -exec convert {} {}.jpg \;

Finding code

grep has many flags that make it a very versatile tool. Some I frequently use are -C for getting Context around the matching line and -v for inverting the match, i.e. print all lines that do not match the pattern. For example, grep -C 5 will print 5 lines before and after the match. When it comes to quickly searching through many files, you want to use -R since it will Recursively go into directories and look for files for the matching string.

Finding shell commands

The history command will let you access your shell history programmatically. It will print your shell history to the standard output. If we want to search there we can pipe that output to grep and search for patterns. history | grep find will print commands that contain the substring “find”.

Lastly, a thing to have in mind is that if you start a command with a leading space it won’t be added to your shell history. This comes in handy when you are typing commands with passwords or other bits of sensitive information. If you make the mistake of not adding the leading space, you can always manually remove the entry by editing your .bash_history or .zhistory.

Directory Navigation

Exercises

PreviousCourse overview + the shellNextEditors (Vim)

Last updated 4 years ago

Was this helpful?

Unlike other scripting languages, bash uses a variety of special variables to refer to arguments, error codes, and other relevant variables. Below is a list of some of them. A more comprehensive list can be found .

In the comparison we tested whether $? was not equal to 0. Bash implements many comparisons of this sort - you can find a detailed list in the manpage for . When performing comparisons in bash, try to use double brackets [[ ]] in favor of simple brackets [ ]. Chances of making mistakes are lower although it won’t be portable to sh. A more detailed explanation can be found .

Writing bash scripts can be tricky and unintuitive. There are tools like that will help you find errors in your sh/bash scripts.

The kernel knows to execute this script with a python interpreter instead of a shell command because we included a line at the top of the script. It is good practice to write shebang lines using the command that will resolve to wherever the command lives in the system, increasing the portability of your scripts. To resolve the location, env will make use of the PATH environment variable we introduced in the first lecture. For this example the shebang line would look like #!/usr/bin/env python.

Functions are executed in the current shell environment whereas scripts execute in their own process. Thus, functions can modify environment variables, e.g. change your current directory, whereas scripts can’t. Scripts will be passed by value environment variables that have been exported using

One of the most common repetitive tasks that every programmer faces is finding files or directories. All UNIX-like systems come packaged with , a great shell tool to find files. find will recursively search for files matching some criteria. Some examples:

Despite find’s ubiquitousness, its syntax can sometimes be tricky to remember. For instance, to simply find files that match some pattern PATTERN you have to execute find -name '*PATTERN*' (or -iname if you want the pattern matching to be case insensitive). You could start building aliases for those scenarios, but part of the shell philosophy is that it is good to explore alternatives. Remember, one of the best properties of the shell is that you are just calling programs, so you can find (or even write yourself) replacements for some. For instance, is a simple, fast, and user-friendly alternative to find. It offers some nice defaults like colorized output, default regex matching, and Unicode support. It also has, in my opinion, a more intuitive syntax. For example, the syntax to find a pattern PATTERN is fd PATTERN.

After pressing Ctrl+R, you can type a substring you want to match for commands in your history. As you keep pressing it, you will cycle through the matches in your history. This can also be enabled with the UP/DOWN arrows in . A nice addition on top of Ctrl+R comes with using bindings. fzf is a general-purpose fuzzy finder that can be used with many commands. Here it is used to fuzzily match through your history and present results in a convenient and visually pleasing manner.

Another cool history-related trick I really enjoy is history-based autosuggestions. First introduced by the shell, this feature dynamically autocompletes your current shell command with the most recent command that you typed that shares a common prefix with it. It can be enabled in and it is a great quality of life trick for your shell.

So far, we have assumed that you are already where you need to be to perform these actions. But how do you go about quickly navigating directories? There are many simple ways that you could do this, such as writing shell aliases or creating symlinks with , but the truth is that developers have figured out quite clever and sophisticated solutions by now.

As with the theme of this course, you often want to optimize for the common case. Finding frequent and/or recent files and directories can be done through tools like and . Fasd ranks files and directories by , that is, by both frequency and recency. By default, fasd adds a z command that you can use to quickly cd using a substring of a frecent directory. For example, if you often go to /home/user/files/cool_project you can simply use z cool to jump there. Using autojump, this same change of directory could be accomplished using j cool.

More complex tools exist to quickly get an overview of a directory structure: , or even full fledged file managers like or .

here
test
here
shellcheck
shebang
env
export
find
fd
zsh
fzf
fish
zsh
ln -s
fasd
autojump
frecency
tree
broot
nnn
ranger
Logo./missing-semester - Shell Tools and Scripting - Exerciseszacheller@home:~/blog$