Data Science Pandas Python

How to import existing files with Pandas

Pinterest LinkedIn Tumblr
pandas thumbnail

In this tutorial, we will learn how to import existing files into our current working project using Pandas. We will also learn how to work with file formats like CSV (Comma Separated Values), JSON (JavaScript Object Notation), and Excel.

This tutorial is part three in our three-part series on the fundamentals of Pandas:

  1. Part #1: A simple walk-through with Pandas for Data Science, Part 1 (tutorial from two weeks ago)
  2. Part #2: A simple walk-through with Pandas for Data Science, Part 2 (last week’s tutorial)
  3. Part #3: How to Import existing files with Pandas (today’s tutorial)

Overview

To recapitulate what we have covered in the last two tutorials, we have seen how to create and perform data selection of Pandas objects. We also further explained how to handle missing data and looked at two methods for combining datasets from different sources. Finally, working with groupby and computing functions like aggregate, apply, filter, and transform.
I recommend you have a quick read to further advance your skills in working with tabular data, if you already haven’t.

How to load and save CSV files in Python with Pandas

A CSV file (Comma-Separated Value) is a raw text file which any of your favorite text editors can display the content. The way the files are stored is in a tabular form, where commas separate columns and new lines separate rows. When looking at the structure, the first row contains the names of the columns for the data.

Now let’s see how will we load or save a .csv file. Let’s begin by opening a Python file or a Python script, name it pandas_tutorial.py or  pandas_tutorial.ipynb (A Jupyter notebook), then type in this code snippet and execute your code. 

First start by importing the needed Python library Pandas into our working script. Then read the .csv file from a web server by passing the URL link as a parameter into the pd.read_csv predefined function.

Note: As for the argument filepath_or_buffer, we can either pass in a URL link to the datasets or a path to the file on our local machine.

As for the separator (sep) argument, the comma is utilized since we know the output file’s delimiter. It’s also important to know other delimiters like a semi-colon (“;”) or a tab (“t”) can be substituted instead of a comma (“,”). 


To save the pandas.DataFrame object into a file, we need to call the predefined function .to_csv that writes the object to a comma-separated values (CSV) file. You should also pay close attention to the separator (sep) specified, which will be the field delimiter for the output file.

How to load and save Json Format files in Python with Pandas

If you haven’t already heard of JSON, it stands for JavaScript Object Notation. It’s a widely used lightweight format for data storage and transportation of data between a client and a server. 

It’s also publicly adopted by developers because it’s readable compared to formats like XML (Extensible Markup Language). Yikes. 

Due to its widespread presence and influence on programming, you’ll likely want to learn how to read JSON from a file, server response, or write JSON to a file at some point in your development. 

Trust me, both of these jobs are a breeze with Python. You’ll see.

One of the easiest ways to write your data in the JSON format to a file using Python is to store your data in a dictionary object, then convert the dictionary object into a pandas object (DataFrame).

To save the pandas object as type JSON, we need to call the .to_json predefined function, pass the name of the document in addition to the file type .json at the end.

Above all, reading JSON data from a file is as easy as writing it to a file. Using the predefined function pandas.read_json from Pandas, we can extract and parse the JSON string directly from a file.

How to load and save Excel files in Python with Pandas

Excel is a popular spreadsheet developed by Microsoft for Windows, IOS, Android, and macOS. You must have used it when performing some calculations, make when making your monthly budget list, creating visualizations, etc.

Let’s see how we can save a pandas object in the form of an excel file.

Let’s create a Python dictionary of my favorite football players, with three different pieces of information about the players: their name, gender, and nationality.
From the Python dictionary initialized, let’s convert the dictionary into a pandas object.

When saving the DataFrame as an excel spreadsheet, it’s required to specify the location to save the file and the sheet-name if needed. Also, set the index to False if you don’t want to include the row names.

Now we have seen how to write only to one sheet. The question now is, what if we want to write into multiple sheets. Like, save all my top 3 NBA players into one sheet and top 5 football players into another sheet within the same file? First, we will make a dictionary and create a writer object that will let us write the DataFrame object into excel sheets.

Once we iterate through the keys within the variable sport_sheet (‘Football Players’ and ‘Basketball Players’), we will save the distinct key and value pair into the save file, yet indifferent sheet_name. Then eventually, call writer.save that saves the workbook to disk.

How to load specific sheets in Excel

What if, from the excel spreadsheet, we want to only load all the data in the sheet name, “Football Players”? We need to pass the name of the sheet to the argument “sheet_name”.

Likewise, the process is possible for the Basketball players saved in the sheet_name, “Basketball Players”.

Reading Specific Columns from an Excel File

In most cases, reading all the columns within your spreadsheet isn’t want you to want to achieve. Some columns aren’t of interest to you. You could only access specific columns of interest to you by passing the indexes as a parameter to the argument usecols. Let’s add the parameter to read the columns that correspond to the basketball player’s “Name” and “Nationality”.

You should click on the “Click to Tweet Button” below to share on twitter.

Check out the post on a simple walk-through with Pandas. Click To Tweet

Conclusion

In this post, you discovered how to load and save existing file formats. These formats include CSV, JSON, and Excel. Be sure to explore more options on the type of files you may read and save here.

In the next tutorial, we will explore Matplotlib, a library for visualizing data into pie charts, bar charts, histograms, line plots and whatever kinds of graphs you can imagine.

Do you have any questions about Pandas or this post? Leave a comment and ask your question. I’ll do my best to answer.

To get all access to the source code used in all tutorials, leave your email address in any page’s subscription forms.

Further Reading

We have listed some useful resources below if you thirst for more reading.

Articles

Books

To be notified when this next blog post goes live, be sure to enter your email address in the form!

Write A Comment