Fastest Way To Write Huge Data In File Using Python

The need for the fastest way to write huge amounts of data into a file is increasing rapidly. Nowadays, we require data in every domain. The domains like machine learning, deep learning, data science, and data analytics are based on the data. The models predict the results after analyzing a large amount of data. The accuracy of models is based on the size of the data. The data can be in any form and structure. This data can be either structured or in an unstructured format. The information can be represented in the form of digital or analog.

Structured data is available in tabular form, spreadsheet, CSV, or SQL databases. This type of data is easy to handle. Unstructured data is available in free format. This type of data does not exist in tabular or well-structured form. This article focuses on the fastest methods to write huge amounts of data into a file using Python code.

Why do we need to Import Huge amounts of data in Python?

Data importation is necessary in order to create visually appealing plots, and Python libraries like Matplotlib and Plotly are capable of handling large datasets. One common step in working with text data for Natural Language Processing (NLP) applications like sentiment analysis, language mode­ling, or topic modeling is importing large text. These applications heavily depend on NLP techniques.

In real-time cases, the need may arise to import and process incoming data in streaming applications. These types of applications generate continuous data over time. To avoid this challenge, distributed computing frameworks like Dask, PySpark, or MPI are implemented. Utilizing such frameworks often involves distributing large datasets across multiple nodes or processes for efficient execution.

Large datasets are used for data analysis, machine learning, or statistical modeling in Python. This step is crucial when conducting various tasks like data cleaning, transformation, and modeling.

The Reason Behind the ‘Slowness’ While Writing Huge Data In a File

In the normal way, huge data in Python refers to data that takes more time to process and big size (many no. of elements present in the data). Therefore, writing this huge data in the fastest way possible in Python is the most important. But first, we need to know the reason behind the slowness. For smaller datasets, the time does not affect the total time to process the code. But for large datasets, it is very important to control the speed of importing. This process of importing may take time because of huge datasets, but we can use different strategies known in the Python language. For example, we can use the Mmap package for normal datasets. If you are dealing with non-textual data, then you can utilize buffered writing. This will make the general process easy. To bypass the problem of importing huge data, we can use the method of parallel processing. Let’s see this approach one by one.

Fastest Way to Write Huge Data in Python

There are different ways to write huge data in Python. In method 1, I implemented the memory-mapped files. For that, I used the mmap module from Python. The file name should be provided in this code to store the huge data. You can add your own data in place of my data. For reference, I implemented the code. You can replace the filename and data and then execute the code.

Mmap Module in Python

import mmap

def write_large_data_to_file(filename, data):
    with open(filename, "wb") as file:
        with mmap.mmap(file.fileno(), 0, access=mmap.ACCESS_WRITE) as mapped_file:
            mapped_file.write(data)

huge_data = b"Your huge data goes here..." 
write_large_data_to_file("large_data_file.bin", huge_data)

The simple execution of the mmap module for writing huge data in Python The mmap() function requires the filename as an argument where we are going to store the data. Then, the .write() is a function used to write the mapped data into the file. Then, you can customize the code by inserting your data. Let’s see the result.

Here, you need to enter the filename and the data in the code. In this way, you can write your huge data into a desired file.

Buffered Writing for Non-textual Data with Binary Numbers

When we are dealing with non-textual data and we want to write this data into the file, we can use this technique. The sample code for this data is very simple. We are writing the data in binary format instead of actual text, which is way faster than any other technique. Let’s see the sample code for more understanding.

def write_large_data_to_file(filename, data, chunk_size=8192):
    with open(filename, "wb", buffering=chunk_size) as file:
        for chunk in data:
            file.write(chunk)

In this sample code, we are initializing a function that takes filename, data, and chunk size as arguments. Then, we can write our huge data into the file using the .write() function. The data is entered in the form of binary instead of text.

You can insert your data and pass it as an argument to the given function.

Parallel Processing to Write Huge Data in Python

There is one more package in Python that is used to implement parallel processing, called the Multiprocessing module. The purpose of using the multiprocessing module in this code is to increase the time required to copy your data to the required file. We are using this to import and export large amounts of data into the files.

import multiprocessing

def write_chunk_to_file(args):
    filename, chunk = args
    with open(filename, "ab") as file:
        file.write(chunk)

def write_large_data_to_file_parallel(filename, data, num_processes=4):
    pool = multiprocessing.Pool(processes=num_processes)
    chunk_size = len(data) // num_processes
    chunked_data = [data[i:i + chunk_size] for i in range(0, len(data), chunk_size)]
    args_list = [(filename, chunk) for chunk in chunked_data]
    pool.map(write_chunk_to_file, args_list)
    pool.close()
    pool.join()

In this sample code, we imported the multiprocessing module and also initialized two different functions to implement the method. The first function helps write the data into the file. Then, we described another function that implements the process of multiprocessing. In this sample code, we are implementing 4 processes at the same time. This function will write the data into the file.

In this way, you can enter your own data and filename to implement the sample code.

All three methods are the fastest way to write huge data in Python. For normal text data, you can prefer the first method of the mmap module. For non-textual data, use the buffered method, and for large amounts of mixed data, we can use multiprocessing.

Summary

Data is a basic part of every module of machine learning, data science, data analysis, and deep learning. There are a few reasons behind the slowness of writing/ importing huge data in Python. This article focuses on such methods that are used to write huge data in the fastest way. We can use different modules like mmap, multiprocessing, and some methods like a buffered method to write different kinds of huge data in the file format. I hope you will enjoy this article!

References

Do read the official documentation on the multiprocessing module.

Here’s stackover