Does DevSecOps Engineer need programming skills? What is the value of utilising Python for security purposes? What you need to learn to automate security at scale? This article will answer these questions.
Introduction
Nowadays, DevSecOps concept is not only a buzzword but is a commonly observed approach across various companies where security matters — especially in bigger technology or financial companies. DevSecOps can be explained as an approach to DevOps where Security steps are placed after the Development activities such as programming but before deploying the code and making projects Operational. In the following chapters of this article I will present why Python is useful for any engineer working in the DevSecOps area and what is worth learning in my opinion. In a number of companies there may not be a dedicated DevSecOps position but it can be covered by DevOps or Security Engineering positions. However, this article may still be relevant for any engineer who would like to apply security controls at large scale using automated solutions developed with Python.
If you’re new to DevSecOps concepts, they are greatly explained in the article — AppSec: SecDevOps or DevSecOps? Do We Need to Choose? Guide to the What and the Why.
Why Python?
You can ask me why I’ve chosen Python and not JavaScript, Ruby, Go or any other scripting language that could be used for DevSecOps Engineer? From my experience, Python is practically everywhere… many security tools are implemented with this language and it has a number of third party client modules for interacting with various platforms. Furthermore, it’s relatively easy and fast to develop any custom integration or write a small piece of code to automate some task. Let’s imagine that you would like to propagate vulnerabilities identified by some third party security solution to a central vulnerability management platform but there is no dedicated integration mechanism. If they both have HTTP APIs this goal is rather straightforward to achieve with Python!
In general, Python is great to automate common tasks or develop custom CLI tools utilised in CI/CD pipelines. For example it’s common to implement tasks of downloading and propagating data from one data source to another location. Another example of a common use case is to write a CLI tool responsible for security checks that will be placed in CI/CD. Moreover, this language is widely used not only in the cybersecurity area but also in other areas such as web applications development, data science, testing or finances.
Few examples of popular security projects developed with Python:
- Scapy — Interactive packet manipulation program and library,
- OWASP DefectDojo — Vulnerability Management platform,
- Semgrep — Static Application Security Testing tool,
- pwntools—exploit development library used especially for binary exploitation during CTFs,
- requests — simple, yet elegant, HTTP library.
1. Interacting with APIs
In cybersecurity it’s common to use web based platforms to perform various security related activities. For example, many vendors providing software composition analysis and open-source security management uses web application to track dependencies across projects. They also provide a number of integrations mechanisms with popular services such as Jira, Github, Bitbucket etc. However, there are cases when the integration is not existing because the service that you’d like to integrate is not so popular and the vendor has no business case for that or the service is developed by you. In such cases you could say that nothing can be done and you need to take a few bucks out of your company’s pocket and buy an expensive but suitable solution… But you can do it better and save some money for you or the company by integrating those services via API through a custom tool!
Usually, web applications provide HTTP API for programming interactions, especially modern ones. In those cases you can use Python and Requests library. Requests is an elegant and simple HTTP library for Python, built for human beings.
How simple is that? It’s that simple to retrieve information about current user from GitHub API:
import requests
headers = {"Authorization": "Bearer YOUR_API_TOKEN"}
r = requests.get('https://api.github.com/user', headers)
user_dict = r.json()
print(user_dict)Above code is just an example to present you that interacting with API might be simpler than you think.
Also, in many cases you can find a Python clients for common API services on GitHub. For example, if you’d like to interact with Jira to manage some tickets you can use Jira Python client.
2. Processing JSON, XML and CSV formats
Another very useful knowledge is related with processing popular data formats used by various application. Modern web applications will use mostly JSON format for interactions via API. Older services or the ones associated with Microsoft may use XML format to exchange data. For simple data analysis it can be helpful to save results in JSON or CSV in the file and use the file for further processing later.
JSON can be processed in Python with built-in json library. For example, the following code can load JSON data from string to Python dict, modify it and save as a result in the file.
import json
json_string = '{"key": "value"}'
json_dict = json.loads(json_string)
json_dict["key"] = "value2"
with open("output.json", "w") as f:
json.dump(json_dict, f)For XML and especially HTTP processing, I can recommend Beautiful Soup library. It provides methods to parse, process and modify XML tree in a convenient way. This library is very useful for scraping data from web when HTTP API is not available.
For data stored in Comma Separated Values (CSV) format, there is a built-in csv library. This library is useful when you work with data sheets and performing some basic data analysis which can be imported by other tools.
3. Working with processes
With Python, it’s possible to run other binaries that are normally executed via Command Line Interface (CLI). Furthermore, Python allows to obtain the output and return code from such program. In most cases, a subprocess — a bult-in library can be recommended. For example, the following Python code can be used to run Semgrep from Python script, process JSON output and print data related with identified findings.
import json
import subprocess
args = args = ["semgrep", "--json", "--config=auto", "."]
process = subprocess.run(
args,
check=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE
)
output = json.loads(process.stdout)
results = output.get("results")
for result in results:
rule_id = result.get("check_id")
severity = result.get("extra").get("severity")
affected_path = result.get("path")
print(f"Rule: {rule_id}")
print(f"Severity: {severity}")
print(f"Affected Path: {affected_path}")
print("----------------------")
print(f"Affected Path: {affected_path}")It’s a simple example presenting how to execute a chosen program with Python, obtain output and process that data. If you’re curious about the results, I ran this code against Vulnerable Rest API project. It’s an intentionally vulnerable project presenting OWASP TOP 10 vulnerabilities. The output is presented below:
4. Working with files and directories
Working with files and directories is a fundamental aspect of programming. From time to time, it’s useful to save data in a file, process given files or listing directories. On the following snippets, I will present a few basic tricks in Python:
The following code presents opening and reading the content of the chosen file:
with open("file.txt", "r") as f:
content = f.read()
print(content)To save some data in a file, the following code could be used:
content = "Hello, World!"
with open("file.txt", "w") as f:
f.write(content)Furthermore, with Python it’s also easy to list directory content. The presented os.listdir returns a list of files and directories inside the directory specified in an argument:
import os
dir_content = os.listdir("path/to/directory")
print(dir_content)Those examples are meant to present the fact that Python can be easily used for various files related operations. More in-depth information about files and directories can be found official Python documentation — File and Directory Access.
5. Creating CLI utilities
Security engineers may often see use cases for creating a simple CLI tools to perform some small tasks. Those tasks may be related with obtaining some vulnerability metrics, running security scans or actually anything which can be automated. Often, those tools may accept different input and behave in a different way based on this input. For these cases argparse is a pretty good library.
The argparse module makes it easy to write user-friendly command-line interfaces. The program defines what arguments it requires, and argparse will figure out how to parse those out of sys.argv. The argparse module also automatically generates help and usage messages. The module will also issue errors when users give the program invalid arguments.
To present argparse in action, let’s extend our previous example code and add there a mechanism for accepting arguments from the user. The following code takes --path and --config parameters from the user. It also has a --help command to present basic information how the script can be used.
import argparse
import json
import subprocess
# define a parser object
parser = argparse.ArgumentParser(
prog='Semgrep CLI Wrapper',
description='It executes Semgrep SAST tool with pre-defined parameters and process JSON output')
# add arguments
parser.add_argument(
'-p', '--path', help="a path to the scanned location", default="."
)
parser.add_argument(
'-c', '--config', help="a config used by Semgrep", default="auto"
)
args = parser.parse_args()
config = args.config
path = args.path
args = ["semgrep", "--json", f"--config={config}", path]
# the rest of the code goes below
...I saved the script as sast_cli.py and executed it with --help . The library automatically creates the output for this parameter and it can be observed below:
Now, let’s run our CLI tool with --path and --config to initiate the scan:
As you can see argparse can be pretty helpful for creating CLI tools.
6. Developing basic web applications
What? Security Engineer developing web apps? I could become a Backend or Fullstack engineer if I’d like to create web apps…
In fact, this might be a rarely used skill and web apps development is not a main domain of security engineers. However, from my personal experience and based on many open-source projects, web application development might be a highly appreciated skill. Sometimes, a basic application need to be created for internal purposes to manage custom processes. Furthermore, many security solutions have UI implemented as a web application. A great example is Security Drone developed internally in Revolut which has an API server used for managing security scans. More about Security Drone can be found in the following article:
In terms of web application technologies in Python it’s worth learning at least one of the following frameworks:
- Flask
- FastAPI
- Django
If you know or want to learn some other technology it’s also good for you. This small list is a suggestion but not a must-have. Learning one web application framework will make it easier to learn another one if needed in future.
Summary
In this article, I presented several areas and libraries that can be useful for any Security Engineer. In my opinion these libraries are valuable to learn as I’m using them very often in my projects. Let’s quickly summarise presented libraries and frameworks with their use-cases:
- requests — interacting with HTTP API,
- subprocess — executing commands from Python script,
- json — processing JSON content,
- Beautiful Soup — parsing HTML and other XML content,
- csv — processing CSV,
- argparse — creating CLI tools that can accept user-provided parameters,
- Flask, FastAPI, Django — web applications development
