# 11 must-know Python Pandas tips and tricks for data scientists

Adapted from the original source: [Read the full article here](https://dataqubed.io/blog/pandas-tips-and-tricks-for-data-scientists/).


Pandas is considered the most widely used tool for data manipulation, filtering, and wrangling. It comes with enormous features and functionalities designed for fast and easy data analytics. As a data scientist, I use its features on a daily basis, and in this post, I’d love to share with you some of the tricks of pandas.

I hope that this crafted list of features serves as a quick reference and gives you a good starting point for learning Pandas. The listed features come in no specific order, so no conclusions can be made on the order.

I will be using the NVDA ticker price data set for this post. So before going into the feature lists, I’ll share with you a few lines of code to download the data locally.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1706247367147/77d9c9e5-3598-4c6e-8361-889f3740408d.png align="center")

### Pull the NVDA exchange pricing data

We will be using Yahoo! Finance's API to download some market data. For that, we will be using the yfinance Python package. Follow [this link and install yfinance](https://pypi.org/project/yfinance/). I installed it using the following single line of code:

```python
!pip install yfinance
```

```python
import os
import numpy as np
import pandas as pd
import pickle
import yfinance as yf
```

Let's first pull the historical NVDA exchange rate from the Yahoo! Finance's API.

```python
def get_yf_data(ticker, period = "1mo"):
    '''Pull and cache yfinance data into a dataframe. Caching is important for avoiding rate limits.'''
    pkl_file_name = ticker.replace(" ", "_") + "_" + period 
    cache_path = '{}.pkl'.format(pkl_file_name).replace('/','-')
    try:
        # Check if the pickle file is in the working directory.
        f = open(cache_path, 'rb')
        ticker_df = pickle.load(f)   
        print('Loaded {} from cache'.format(ticker))
    except (OSError, IOError) as e:
        # The data is not locally avaiable. Download it and store it in a form of pickle file. 
        print('Downloading {} from yfinance'.format(ticker))
        ticker_df = yf.download(ticker, period=period)
        ticker_df.to_pickle(cache_path)
        print('Cached {} at {}'.format(ticker, cache_path))
    return ticker_df

# Pull NVDA and AAPL price from exchange data
NVDA_price= get_yf_data("NVDA", period="10d")

# If you would like to download more tickers you try the following:
NVDA_price= get_yf_data("NVDA APP TSLA", period="1mo")
# The above will download the Nvidia, Apple and Tesla share prices. 

NVDA_price.tail()
```

Now that we have the data ready at hand, let's have a look at my selection of Pandas tips and tricks.

### Number 11: Apply aggregations on DataFrame and Series

```python
NVDA_price.agg(['mean', 'min', 'max'])

NVDA_price.High.agg(['mean', 'min', 'max'])
```

`.describe()` provides the same functionality but `.agg()` is more flexible as far as I can tell.

```python
NVDA_price.describe()
```

### Number 10: All you need is pandas\_profiling

`pandas_profiling` is a Python library mainly used for initial data exploration. A running example:

```python
# Make sure that you have the pandas_profiling installed.
# To install pandas_profiling run: !pip install pandas_profiling

import pandas_profiling

NVDA_price.shape
# Chart the NVDA pricing data
pandas_profiling.ProfileReport(NVDA_price)
```

This helps you with a few lines of code and time-saving tricks when exploring the data. The `pandas_profiling` library comes with more features than the one listed above. Interested readers are referred to the official website.

The package provides an extensive exploratory data analysis (EDA) that includes reporting on different types of correlations, such as Pearson's r and Phik coefficient (φk). Phik (φk) is a new correlation coefficient that works consistently between categorical, ordinal, and interval variables. Phik can capture non-linear dependency and revert to the Pearson correlation coefficient in the case of the bivariate normal input distribution. Read more [here](https://phik.readthedocs.io/en/latest/index.html).

### **Number 9: Easy way to convert data to**`int` type

This is a simple but useful trick! When converting the data into `int` type, the most common approach is to use `.astype('int')`. However, if there are strings or other invalid types, you will face an error, and the conversion will fail. An alternative is to use the [`pd.to`](http://pd.to)`_numeric()` command which will handle the errors. You may try this for yourself.

### Number 8: Break up strings in multiple columns

A simple trick for splitting a column into multiple columns using a split function:

```python
NVDA_price['column_name'] = NVDA_price['column_name'].str.split(" ", expand=True)
```

### Number 7: .str

I’d like to show you how awesome `.str` is with `Pandas Series`. Check the following piece of code:

```python
# Covert the Data column to str.
NVDA_price['Date_str'] = NVDA_price.index.astype(str)
# And take the first 4 elements of the resulting string. 
NVDA_price['Year_str'] = NVDA_price['Date_str'].str[:4]
```

It's awesome how they make this bit of magic work.

### Number 6: Concatenate strings

It’s so easy with the pandas to create a new column by combining two existing columns:

```python
NVDA_price['ts'] = NVDA_price.index.values.astype(np.int64) // 10 ** 9 
NVDA_price['price_ts'] = NVDA_price["Open"].map(str) + " - " + NVDA_price["ts"].map(str)
NVDA_price['price_ts'].head()
```

and to get them back again:

```python
NVDA_price['price'], NVDA_price['ts'] = zip(*NVDA_price.price_ts.str.split(' - ', 1))
NVDA_price[['price_ts', 'price', 'ts']].head()
```

### Number 5: Percentage of missing data

To get the percentage of missing data in columns and print them in descending order:

```python
# gives % of missing data in columns!
(NVDA_price.isnull().sum()/NVDA_price.isnull().count()).sort_values(ascending=False)
```

### Number 4: Highly correlated columns

Let’s assume that we want to build a predictive model and we would like to know which columns are highly correlated (bigger than 0.10) to the column `Volume`!

```python
corr_coeff = NVDA_price.corr()['Volume'].abs().sort_values()
corr_coeff[corr_coeff > 0.10]
```

### Number 3: Apply and lambda functions

One of the most awesome things that you can do with pandas is that you can store objects such as `networkx` graphs in them. Then you can easily `.apply()` the metrics and algorithms on each graph and see the tabulated results in a nice pandas dataframe. You can also easily `.apply()` custom functions or `lambdas` to modify all objects at once. The index then can be used to keep track of everything and to keep results near their original objects.

In the example let’s assume that you have multiple networks from different years. We will make a graph at `random`.

```python
import networkx as nx
import itertools
import random

years, graphs = list(zip(*[(year, nx.barbell_graph(random.randint(2,10),random.randint(2,50))) for year in range(2010,2019)]))
df = pd.DataFrame({"graph": graphs}, index=years)

df["number_of_nodes"] = df.graph.apply(nx.number_of_nodes)
df["number_of_edges"] = df.graph.apply(nx.number_of_edges)
df["transitivity"] = df.graph.apply(nx.transitivity)
```

```python
def nodes_odd_degree(g):
    # Calculate list of nodes with odd degree
    nodes_odd_degree = [v for v, d in g.degree() if d % 2 == 1]
    return nodes_odd_degree

def len_odd_node_pairs(g):
    # Compute all pairs of odd nodes and return the length of list of tuples!
    odd_node_pairs = list(itertools.combinations(nodes_odd_degree(g), 2))
    return len(odd_node_pairs)
```

```python
df["nodes_odd_degree"] = df.graph.apply(nodes_odd_degree)
df["len_nodes_odd_degree"] = df["nodes_odd_degree"].apply(len)
df["len_odd_node_pairs"] = df.graph.apply(len_odd_node_pairs)

# And now print a selection of columns:
df[["number_of_nodes", "number_of_edges", "transitivity", "nodes_odd_degree", "len_nodes_odd_degree", "len_odd_node_pairs"]]
```

### Number 2: drop\_duplicates vs unique

Let’s assume that you have a column with millions of entries and you need to find the unique set of values It. `df.column_name.drop_duplicates(keep="first", inplace=False)` is a lot faster than `df.column_name.unique()`. Can be tested using `%timeit`.

The `keep="first"` option drops all the duplicates except for the first occurrence.

### Number 1: Ufuncs

As explained [here](https://towardsdatascience.com/pandas-tips-and-tricks-33bcc8a40bb9) Pandas `Ufuncs` is much better than `.apply` command.

## Wrap-Up

This post should give you an idea of some Pandas tricks commonly used by data scientists. The list aims to make your code more efficient.

In my next post, I'll share some basic concepts you need for working with data in `PySpark`. The post will cover topics such as reading and writing data from CSV files, along with some basic statistical analysis and visualization of data.

---

### Notes:

1 - The official panda[s cheat sheet can be downloaded from he](https://learnpython.com/course/python-data-science?icn=blogTraffic&ici=link-ArticleBottom)[re](https://github.com/pandas-dev/pandas/blob/master/doc/cheatsheet/Pandas_Cheat_Sheet.pdf).

2- Link to the [notebook of this tutorial](https://github.com/mdavarynejad).

### Refernces:

1 - [Data School](https://www.youtube.com/watch?v=FxrGx8cPJlQ&t=33s)

2 - [Lukas Erhard](https://gist.github.com/luerhard/703716567879afefa59c06e5af48c371)

---

Please leave a message below if you would like to share your thoughts or if you like to share your tricks on Pandas.