Obtain All NASDAQ & NYSE Stocks

The website nasdaqtrader.com maintains a list of all stock symbols for stocks on the NASDAQ and NYSE stock exchanges. The data is provided in 2 text files on their ftp site at ftp.nasdaqtrader.com/symboldirectory. The file nasdaqlisted.txt contains the stocks symbols on the NASDAQ stock exchange while the file otherlisted.txt contains the stocks on the NYSE stock exchange. We can download these files using the python script seen below:

from ftplib import FTP

directory = 'symboldirectory'
filenames = ('otherlisted.txt', 'nasdaqlisted.txt')

ftp = FTP('ftp.nasdaqtrader.com')
ftp.login()
ftp.cwd(directory)

for item in filenames:
    ftp.retrbinary('RETR {0}'.format(item), open(item, 'wb').write)

ftp.quit()

In a Linux console, I can see the first few lines of the nasdaqlisted.txt file using head nasdaqlisted.txt.

Symbol|Security Name|Market Category|Test Issue|Financial Status|Round Lot Size|ETF|NextShares
AABA|Altaba Inc. - Common Stock|Q|N|N|100|N|N
AAL|American Airlines Group, Inc. - Common Stock|Q|N|N|100|N|N
AAME|Atlantic American Corporation - Common Stock|G|N|N|100|N|N
AAOI|Applied Optoelectronics, Inc. - Common Stock|G|N|N|100|N|N
AAON|AAON, Inc. - Common Stock|Q|N|N|100|N|N
AAPL|Apple Inc. - Common Stock|Q|N|N|100|N|N
AAWW|Atlas Air Worldwide Holdings - Common Stock|Q|N|N|100|N|N
AAXJ|iShares MSCI All Country Asia ex Japan Index Fund|G|N|N|100|Y|N
AAXN|Axon Enterprise, Inc. - Common Stock|Q|N|N|100|N|N

The otherlisted.txt file varies slightly, and the first few lines are show below:

ACT Symbol|Security Name|Exchange|CQS Symbol|ETF|Round Lot Size|Test Issue|NASDAQ Symbol
A|Agilent Technologies, Inc. Common Stock|N|A|N|100|N|A
AA|Alcoa Corporation Common Stock |N|AA|N|100|N|AA
AAC|AAC Holdings, Inc. Common Stock|N|AAC|N|100|N|AAC
AADR|AdvisorShares Dorsey Wright ADR ETF|P|AADR|Y|100|N|AADR
AAMC|Altisource Asset Management Corp Com|A|AAMC|N|100|N|AAMC
AAN|Aaron's, Inc. Common Stock|N|AAN|N|100|N|AAN
AAP|Advance Auto Parts Inc Advance Auto Parts Inc W/I|N|AAP|N|100|N|AAP
AAT|American Assets Trust, Inc. Common Stock|N|AAT|N|100|N|AAT
AAU|Almaden Minerals, Ltd. Common Shares|A|AAU|N|100|N|AAU

This is great because now with the use of pandas and pandas-datareader I can obtain data for any stock listed in these files. First, we should import all of the python packages that we want to used and create a pandas data frame for the data in each of the text files. Creating the dataframes is easy, because the text files are delmininated using a pipe character (|). We can use the pandas read_csv() function to read the data. Just be sure to change the deliminator from a comma (,) to a pipe (|). See below:

import pandas_datareader.data as web
from datetime import datetime, timedelta
import pandas as pd
import pickle

# Create pandas dataframes from the nasdaqlisted and otherlisted files.
nasdaq_exchange_info = pd.read_csv('nasdaqlisted.txt', '|')
other_exchange_info = pd.read_csv('otherlisted.txt', '|')

Now, lets say we want to find all stock symbols that are beating the S&P 500 by more than 10% over the last 365 days, and store the last 365 days worth of data for those stock tickers into a python pickle for later analysis. We can use pandas-datareader for this and obtain the stock ticker ‘SPY’, which is an ETF that closly follows the S&P. We will use the data from ‘SPY’ to compare to the other stock symbols.

end = datetime.now()
start = end - timedelta(days=365)

valid_sources = ('morningstar', 'iex')
close_string = 'close'
source = valid_sources[1]

spy = web.DataReader('SPY', source, start, end)
spy_percent_change = (spy.iloc[-1][close_string] - spy.iloc[0][close_string]) / \
                      spy.iloc[0][close_string] * 100

Currently, pandas-datareader has the ability to obtain data from many sources. I like using Morningstar, but IEX seemed to work best for me when executing this script. In the portion of python script above, we obtain data for SPY and calculate the percent chane over the last 365 days.

Next, we create a python dict, iterate over the stock symbols in the nasdaqlisted.txt file, and store the stock symbol and data for that symbol into the dict if the percent change for that symbol is more than 10% higher than the percent change of SPY.

Notice that we use try / except to catch any errors. I noticed that sometimes IEX wouldn’t respond, or IEX wouldn’t know what to do with a certain stock symbol. When this happens, I just print the exception to the console and move to the next symbol. I also filter out any stocks that are under $10 in value. I find these stocks to be too risky and unstable. After collecting all of the stock symbols into a dict, I then store that dict to a pickle file for later analysis.

valid_symbols_dict = dict()
for symbol in nasdaq_exchange_info['Symbol']:
    try:
        print('Testing Symbol {0}'.format(symbol))
        f = web.DataReader(symbol, source, start, end)
        last_close = f.iloc[-1][close_string]
        if last_close > 10.0:
            start_close = f.iloc[0][close_string]
            percent_change = (last_close - start_close) / start_close * 100
            if percent_change - spy_percent_change > 50.0:
                valid_symbols_dict[symbol] = f
                print(valid_symbols_dict.keys())
    except Exception as e:
        print(e)

pickle.dump(valid_symbols_dict, open('nasdaq.p', 'wb'))

Finally, we do the same thing for the symbols in the otherslisted.txt file.

valid_symbols_dict = dict()
for symbol in other_exchange_info['ACT Symbol']:
    try:
        print('Testing Symbol {0}'.format(symbol))
        f = web.DataReader(symbol, source, start, end)
        last_close = f.iloc[-1][close_string]
        if last_close > 10.0:
            start_close = f.iloc[0][close_string]
            percent_change = (last_close - start_close) / start_close * 100
            if percent_change - spy_percent_change > 50.0:
                valid_symbols_dict[symbol] = f
                print(valid_symbols_dict.keys())
    except Exception as e:
        print(e)

pickle.dump(valid_symbols_dict, open('other.p', 'wb'))

Using the data available from nasdaqtrader.com and the pandas-datareader python package, it is possible to obtain all stock symbols on the NASDAQ and NYSE stock echange, obtain data for those stock symbols, and analyze the resulting data.

Leave a Reply

Close Menu