JSON Processing with Shell Scripts

Introduction to JSON Processing

JSON (JavaScript Object Notation) is a lightweight data interchange format that's easy to read and write for humans and easy to parse and generate for machines. This tutorial will cover various techniques and tools to process JSON data using shell scripts.

Reading JSON Files

Reading JSON files in shell scripts can be accomplished using tools like jq, which is a powerful command-line JSON processor.

Basic JSON Reading

#!/bin/bash

# Read and display JSON data from a file
FILE="data.json"

jq '.' "$FILE"
                

This script reads a JSON file and prints its contents in a formatted manner.

Extracting Specific Fields

One of the main tasks when processing JSON is extracting specific fields. This can be easily done with jq.

Extracting a Single Field

#!/bin/bash

# Extract a specific field from a JSON file
FILE="data.json"
FIELD=".name"

jq "$FIELD" "$FILE"
                

This script extracts and prints the value of the name field from a JSON file.

Extracting Multiple Fields

#!/bin/bash

# Extract multiple fields from a JSON file
FILE="data.json"
FIELDS=".name, .age"

jq "{$FIELDS}" "$FILE"
                

This script extracts and prints the values of the name and age fields from a JSON file.

Manipulating JSON Data

In addition to extracting fields, you may need to manipulate JSON data. This can involve modifying values, adding new fields, or deleting existing fields.

Modifying JSON Fields

#!/bin/bash

# Modify a field in a JSON file
FILE="data.json"
FIELD=".name"
NEW_VALUE="\"John Doe\""

jq "$FIELD = $NEW_VALUE" "$FILE"
                

This script modifies the value of the name field in a JSON file.

Adding New Fields

#!/bin/bash

# Add a new field to a JSON file
FILE="data.json"
NEW_FIELD=".city"
NEW_VALUE="\"New York\""

jq "$NEW_FIELD = $NEW_VALUE" "$FILE"
                

This script adds a new field city with the value New York to a JSON file.

Deleting Fields

#!/bin/bash

# Delete a field from a JSON file
FILE="data.json"
FIELD=".age"

jq "del($FIELD)" "$FILE"
                

This script deletes the age field from a JSON file.

Advanced JSON Processing

Advanced JSON processing involves combining multiple operations to handle complex data structures and perform sophisticated data manipulations.

Filtering JSON Data

#!/bin/bash

# Filter JSON data based on a condition
FILE="data.json"
CONDITION=".age > 30"

jq "select($CONDITION)" "$FILE"
                

This script filters the JSON data to include only those entries where the age field is greater than 30.

Combining JSON Data

#!/bin/bash

# Combine two JSON files into one
FILE1="data1.json"
FILE2="data2.json"

jq -s '.[0] * .[1]' "$FILE1" "$FILE2"
                

This script combines two JSON files into one, merging their contents.

Converting JSON to Other Formats

#!/bin/bash

# Convert JSON data to CSV format
FILE="data.json"
FIELDS=".name, .age, .city"

jq -r "[.name, .age, .city] | @csv" "$FILE"
                

This script converts JSON data to CSV format, extracting the specified fields.

Conclusion

Processing JSON data with shell scripts is a valuable skill that enables the automation of data handling tasks. By mastering the techniques and tools covered in this tutorial, you can efficiently extract, manipulate, and transform JSON data in your shell scripts.