Use Python to extract URLs to HTML-format SEC filings on EDGAR

[Update on 2020-06-26] Eduardo has made a significant improvement to the code. Now you can specify a starting date and download the index file during the period from that starting date to the most recent date. I expect it to be very useful for many readers of my website. Eduardo has kindly shared the code in the comment. Thank you, Eduardo!

[Update on 2019-08-07] From time to time, some readers informed that the first-part code seemingly stopped at certain quarters. I don’t know the exact reason (perhaps a server-side issue). I never encountered the issue. I would suggest that you just try again later. I also share a Dropbox link from which you can download the first-part results (as of 2019-08-07; 2.4GB) in the CSV format (link). Please note—as I have explained in my original post, the URL contained in the downloadable CSV is not the URL to the HTML-format filing; it is just the URL to an index page. You need to select your sample and go through the second-part code to get the URL to the HTML-format filing.

[Original post] I wrote two posts to describe how to download TXT-format SEC filings on EDGAR:

Although TXT-format files have benefits of easy further handling, they are oftentimes not well formatted and thus hard to read. A HTML-format 10-K is more pleasing to eyes. Actually, SEC also provides the paths (namely, URLs) to HTML-format filings. With the path, we can open a HTML-format filing in a web browser, or further download the filing as a PDF.

There remain two parts in the Python code. In the first part, we need download the path data. Instead of using master.idx in the above two posts, we need use crawler.idx for this task. The path we get will be a URL like this:

Note that the path we get is a URL to an index page, not a URL to the HTML-format 10-Q in this example. To get the direct URL to the HTML-format 10-Q, we have to go one-level deeper. The second part of the Python code is used to go that deeper and extract the direct URL to the main body of the Form (the URL embedded in the first row in more than 99% cases). The code also extracts such information as filing date and period of report on the index page. The code writes the output (including filing date, period of report and direct URL) in log.csv. The following is an output example—the first URL is the path we get in the first part of the code; the second URL is the direct URL to the HTML-format Form.

The first part of the code:

The first part of the code generates a dataset of the complete path information of SEC filings for the selected period (in both SQLite and Stata). Then, you can select a sample based on firm, form type, filing date, etc. and feed a CSV file to the second part of the code. The feeding CSV should look like this:

The second part of the code:

Please note:

  1. Please use Python 3.x.
  2. Please install all required modules such as Selenium.
  3. The second part of the code uses Selenium. There are other ways to do the job, e.g., using BeautifulSoup.
  4. The second part of the code only output the direct URL to the HTML-format filing; it does not download the filing.
This entry was posted in Python. Bookmark the permalink.

35 Responses to Use Python to extract URLs to HTML-format SEC filings on EDGAR

  1. sara says:

    Hi Kai,
    Thank you very much for your sharing. I am a new Pythoner. Your posts really help me a lot. I was able to run the first part of the code. For the second part of the code, it also ran and the output file is a log.csv. As you said, the csv file contains the direct URL to the HTML-formating filing. Do you know what code can I use to get the HTML-formating filing directly? To make it clear, how can I save each filing as a HTML-formating file automatically? Thanks!

    • Kai Chen says:

      Hi Sara, there are many ways to do this. Just google “python download html” or something similar, you will find the solutions (e.g., use requests or urllib module).

  2. Sylvia Li says:

    Thank you for sharing your code. I am very new to python. I am learning python while trying to work on some data scraping from website (SEC, etc). I could run the first part without any issue. However, there are error messages when I try to run the second part, would you mind checking where might went wrong?

    FileNotFoundError Traceback (most recent call last)
    ~\AppData\Local\Continuum\anaconda3\lib\site-packages\selenium\webdriver\common\ in start(self)
    73 close_fds=platform.system() != ‘Windows’,
    —> 74 stdout=self.log_file, stderr=self.log_file)
    75 except TypeError:

    ~\AppData\Local\Continuum\anaconda3\lib\ in __init__(self, args, bufsize, executable, stdin, stdout, stderr, preexec_fn, close_fds, shell, cwd, env, universal_newlines, startupinfo, creationflags, restore_signals, start_new_session, pass_fds, encoding, errors)
    708 errread, errwrite,
    –> 709 restore_signals, start_new_session)
    710 except:

    ~\AppData\Local\Continuum\anaconda3\lib\ in _execute_child(self, args, executable, preexec_fn, close_fds, pass_fds, cwd, env, startupinfo, creationflags, shell, p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite, unused_restore_signals, unused_start_new_session)
    996 os.fspath(cwd) if cwd is not None else None,
    –> 997 startupinfo)
    998 finally:

    FileNotFoundError: [WinError 2] The system cannot find the file specified

    During handling of the above exception, another exception occurred:

    WebDriverException Traceback (most recent call last)
    in ()
    16 start_time = time.strftime(‘%Y-%m-%d %H:%M:%S’, time.localtime())
    —> 18 driver = webdriver.Chrome(‘./chromedriver’)
    20 try:

    ~\AppData\Local\Continuum\anaconda3\lib\site-packages\selenium\webdriver\chrome\ in __init__(self, executable_path, port, chrome_options, service_args, desired_capabilities, service_log_path)
    60 service_args=service_args,
    61 log_path=service_log_path)
    —> 62 self.service.start()
    64 try:

    ~\AppData\Local\Continuum\anaconda3\lib\site-packages\selenium\webdriver\common\ in start(self)
    79 raise WebDriverException(
    80 “‘%s’ executable needs to be in PATH. %s” % (
    —> 81 os.path.basename(self.path), self.start_error_message)
    82 )
    83 elif err.errno == errno.EACCES:

    WebDriverException: Message: ‘chromedriver’ executable needs to be in PATH. Please see

  3. Bob says:

    Hey Kai,

    thanks for your great tutorial! Its works just fine. But I got a question: Sometimes the loop “for url in urls:” in part 1 needs very long (or does not finish at all, I didnt let it run for a whole night). For example for 2017Q3 or 2011Q4 and some others. Getting all quarterly filings for 2015-2016 works fast and seems to be complete.

    Do you have any idea why this is happening?

    Beste regards!

    • Kai Chen says:

      Hi, thanks for your feedback. It’s an interesting question. I did a quick test on 2017Q3. It turns out to be this line—”lines = requests.get(url).text.splitlines()”—that drags the execution down. I don’t know why, but it appears to be related to the idx file itself.

  4. philip says:

    Hi. Kai
    Thanks you so much for sharing your code.
    I have a question for getting SIC. I opened “crawler.idx”, but I can not find SIC code.
    How can I get SIC code? I inserted “sicloc=lines[7].find(‘SIC’)” , but it does not gives SIC code.
    Thanks in advance

    • Kai Chen says:

      There is no SIC in “crawler.idx”. You can link CIK to SIC using Compustat. In my sample “log.csv”, the first number is an index number generated by Sqlite, and the second number is CIK (not SIC).

  5. annamalai says:

    Hi Kai, Thanks for the code.

    getting an Exception with Chrome Driver. Would you be able to help

    WebDriverException: chrome not reachable
    (Driver info: chromedriver=2.34.522940 (1a76f96f66e3ca7b8e57d503b4dd3bccfba87af1),platform=Windows NT 6.1.7601 SP1 x86_64)

  6. Kim says:

    Hello Kai,

    Thank you very much for sharing this information. This is very helpful.

    I have one question while running the last part of the first code. Specifically, I get an error message when trying to convert the downloads from SQLite to Stata.

    Error message:
    File “C:\Users\user\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\io\”, line 1838, in _dtype_to_default_stata_fmt
    itemsize = max_len_string_array(_ensure_object(column.values))

    SystemError: returned a result with an error set

    When I follow in pandas, it seems to trip over line 1839:

    if itemsize > 244:
    raise ValueError(excessive_string_length_error %

    Would you be able to share your thoughts on this problem? Thanks again for your help!

  7. joe says:

    Hello, kai
    I ran the first part and for some reason it gets stuck running the 4th QTR of year 2011.
    I left it for over 2 hours and it continues to run.
    Do you happen to know what the problem could be?

    • Emma says:

      me too!!
      I cannot use requests, so i change it with urllib.request.urlopen(url) and decode(‘utf-8’) .

      However, in 2011 Q3, i have got an error ‘UnicodeDecodeError: ‘utf-8′ codec can’t decode byte 0xc3 in position 11344121: invalid continuation byte’

      do you have any idea?

  8. ANADRAYA PAI M says:

    “gaierror: [Errno 11001] getaddrinfo failed”
    how to solve this

    getting this error in the code

    ftp = ftplib.FTP(‘’)
    daily_files = [f for f in http://ftp.nlst(‘edgar/daily-index’) if

  9. Tian says:

    Hi Kai,

    Great post! Thank you very much for posting the code.

    One minor suggestion: Instead of using chrome as driver, how about using firefox? The benefit is that we could run it “headless” and not opening the browser each time. What do you think?

    Kind regards,

  10. Erika says:

    This is so helpful – Thank you so much for sharing!

  11. Allen says:

    Hi Kai,

    This is really helpful. Thanks. I was able to run the code the create the database. It takes about 50 mins. How do I update the database without re-run the code from beginning?

    Let’s say now we have data from 1993 to 2019Q3. In Jan 2020, I want to update 2019 Q3 data, since 2019 Q3 has not ended yet, my data may not be full, and then add 2019 Q4 data.

    How do I only pull down 2019 Q3 and Q4 data and add to original database? Also, delete the duplicate data for 2019 Q3 (which I generated before)

    Thanks a lot!

    • Victor says:

      Hi Allen,

      I just downloaded the 2019’s data. I may be able to answer your question.

      Since there are only two quarters, you can manually override the url generation code, up to line 13, with the following:

      history = [(2019, “QTR3”), (2019, “QTR4″)]
      urls = [‘’ % (x[0], x[1]) for x in history]

      Merge with what you have is easy. If you work with stata, you can easily merge the two datasets, and delete the duplicates.

      If you prefer csv, you can convert the sql file to csv by modifying the last few lines of code:

      with engine.connect() as conn, conn.begin():
      data = pandas.read_sql_table(‘idx’, conn)
      data.to_csv(‘edgar_idx_2019_20190801.csv’, index = False)

      To merge with what you have:

      import pandas as pd

      #Load the datasets
      df1 = pd.read_csv(r”old_file.csv”)
      #if you old file is in stata then: df1 = pd.read_stata(r”old_file.dbt”)
      df2 = pd.read_csv(r”new_file.csv”)

      df1 = pd.concat([df1, df2], ignore_index= True)
      #Drop duplicates
      df1 = df1.drop_duplicates()

      #Output the dataset
      df1.to_csv(r”merged.csv”, index = False )

      • Allen says:

        Hi Victor,

        Thanks for your help, this is great. I do not use stata, but I figured out a different way to organize the database. First, run 1993 to 2018Q4 as the base database, then going forward, run calendar year only and merge back to base database. Duplicates won’t be an issue each time when I rerun the calendar year, also, only rerun the calendar year does not take long.

        Thanks for you response anyway. It’s helpful as well 🙂

  12. Lucy says:

    Hi there: when is it that I get an Error! Message instead of Success!

  13. Lucy says:

    Hi Kai,

    I ran your codes today and it doesnt seem to get past 2011 Q3? Is this a problwm with edgar?


  14. Dara says:

    Hi Kai,

    Thank you so much!
    Your posts are of great help to me.

    I used your code to get the html format of SEC filing. But since I need more than 30,000 links, it is getting slower and slower. Is it because SEC has some anti-scraping measure?
    How can I expedite the process?

    If possible, do you provide full html format of SEC filing?

    Many thanks!

  15. Sweet T says:

    Hi Kai!

    Thanks for the posting!!

    I think using selenium here is really an overkill. Noticing that there’s only one “htm” in each index link, we could just use beautifulsoup (like you suggested in this post) to pinpoint the only a-tag that contains the href link to the “htm”, which is what we wanted.

    Lemme know what you think.

  16. Jiadi Chen says:

    Hi Kai,

    Thanks for the posting. I’m working on a text analysis project on EDGAR and run into your website.
    It is very helpful!
    I did some simple words count using both txt link and html link, I find different results for all compaines/files I searched. For example,
    it’s the text file of a 10-K which has word “anticipate” 9 times and “risk” 86 times.
    However, when I try use the html version, I got word “anticipate” 25 times and “risk” 142 times.

    Is there any difference between these two versions of the same file? I understand that you are a cs expert, not a finance person. Please let me know if this happened to you before. Thanks a lot!


    • Kai Chen says:

      I found 39 “anticipate” in txt, while 26 in html. If you compare the hits in both txt and html, you will notice the first 26 hits are exactly the same. The extra hits in txt result from the fact that txt document also includes xml tagging content and thus generates repetition. Txt format has become notoriously unfriendly nowadays and that’s why I would advocate directing our research efforts to html format.

  17. Lucy says:

    Hi Kai,

    Thanks very much for the posting! I am a green hand in this area, the code helped me a lot especially when I try to extract information from SEC. I can successfully generate a 2.76GB edgar_htm_idx.db file but when I run the last part of the first section code, I got the following error info:
    OSError Traceback (most recent call last)
    in ()
    5 with engine.connect() as conn, conn.begin():
    6 data = pandas.read_sql_table(‘idx’, conn)
    —-> 7 data.to_stata(‘edgar_htm_idx.dta’)

    ~/Documents/anaconda3/lib/python3.6/site-packages/pandas/core/ in to_stata(self, fname, convert_dates, write_index, encoding, byteorder, time_stamp, data_label, variable_labels, version, convert_strl)
    1875 write_index=write_index,
    1876 variable_labels=variable_labels, **kwargs)
    -> 1877 writer.write_file()
    1879 def to_feather(self, fname):

    ~/Documents/anaconda3/lib/python3.6/site-packages/pandas/io/ in write_file(self)
    2215 self._write_characteristics()
    2216 self._prepare_data()
    -> 2217 self._write_data()
    2218 self._write_strls()
    2219 self._write_value_labels()

    ~/Documents/anaconda3/lib/python3.6/site-packages/pandas/io/ in _write_data(self)
    2407 def _write_data(self):
    2408 data =
    -> 2409 self._file.write(data.tobytes())
    2411 def _null_terminate(self, s, as_string=False):

    OSError: [Errno 22] Invalid argument

    I am running it on macbook pro with 16GB RAM, I guess it might be related to the size of the stata file it tries to write to? Do you have any idea why it happens and how to fix this issue? Thanks!


  18. Xi Chen says:

    Hi Kai,

    Thanks for your code. I can get results from the first part code.
    However, I fail to get output from the second part code. I got the error in log.csv file. I did not have deeper HTML information when I execute the second part code.

    Start fetching URL to 1414953 28/10/2019 filed on
    Error! 2019-12-15 23:11:23 –> 2019-12-15 23:11:25

    Start fetching URL to 1414953 28/10/2019 filed on
    Error! 2019-12-15 23:11:23 –> 2019-12-15 23:11:25

  19. Eduardo says:

    Hi Kai,

    Your code is all I needed, thanks a lot!!!

    As other mentioned, there are two problems in years 2011 and 2017 that break the program, but as you mentioned, are related with the EDGAR files.

    I used your code and made some changes to get the information from another EDGAR file that have the same information, but instead of having one idx file for the each quarter, have one idx file for each day of the quarter. It takes a litle more time to process but avoid the problems in the 2017 year.

    Also I changed the starting year for a starting date, so anyone can update the SQLite table from the last time the table was updated.

    I am only interested of having the information in SQLite so the code finish there.

    I hope this code can help others who wanted to get all the EDGAR history files.

    Thanks again and Good Luck!


    import datetime

    # You can chose the date you want to starl

    start_date = “2020-02-19”

    start_year = int(start_date[:4])
    current_year =
    years = list(range(start_year + 1, current_year))

    start_quarter = (int(start_date[5:7]) – 1) // 3 + 1

    history = []

    year = current_year – start_year

    # if the start year is different of current year, it takes the remaining quarters of the start year
    if year > 0:

    for i in range(start_quarter, 5):

    history.append((start_year, ‘QTR%d’ % i))

    # if there are year that are in between of the start and current years, it takes all the quarters of that years
    if year > 1:
    quarters = [‘QTR1’, ‘QTR2’, ‘QTR3’, ‘QTR4’]
    history.extend([(y, q) for y in years for q in quarters])

    current_quarter = ( – 1) // 3 + 1

    # finnally it adds the current year quarters
    for i in range(1, current_quarter + 1):

    history.extend([(current_year, ‘QTR%d’ % i)])

    # now history has the total quarters from the starting quarter to current quarter

    base_url = “”
    QTR = “%d/%s/”
    urls = [base_url + QTR % (x[0], x[1]) for x in history]

    # Download index files and write content into SQLite
    import sqlite3
    import requests
    import urllib

    con = sqlite3.connect(‘edgar_htm_idx.db’)
    cur = con.cursor()

    cur.execute(‘CREATE TABLE IF NOT EXISTS idx (conm TEXT, type TEXT, cik TEXT, date TEXT, path TEXT)’)

    for url in urls:
    url1 = url + “index.json”
    file_content = requests.get(url1)
    decoded_content = file_content.json()

    print(‘Pulling files’)

    # for each file in the directory items list, print the file type and file href.

    start_file_date = datetime.datetime(2020, 2, 19)
    for file in decoded_content[‘directory’][‘item’]:

    if file[“name”][0:7] == “crawler”:

    date = file[“name”][8:16]

    file_date = datetime.datetime(int(file[“name”][8:12]), int(file[“name”][12:14]), int(file[“name”][14:16]))

    if file_date >= start_file_date:
    file_url = url + file[‘name’]
    date = file[“name”][8:16]
    print(“Processing working day:”, date, “in Edgar File:”, url[47:56])
    lines = requests.get(file_url).text.splitlines()
    typeloc = lines[7].find(‘Form Type’)
    cikloc = lines[7].find(‘CIK’)
    dateloc = cikloc + 7
    urlloc = cikloc + 20

    records = [tuple([line[:typeloc].strip(), line[typeloc:cikloc].strip(), line[cikloc:dateloc].strip(),
    line[dateloc:urlloc].strip(), line[urlloc:].strip()]) for line in lines[9:]]
    cur.executemany(‘INSERT INTO idx VALUES (?, ?, ?, ?, ?)’, records)

    print(url, ‘downloaded and wrote to SQLite’)
    print(“All files recorded in Database”)

Leave a Reply

Your email address will not be published. Required fields are marked *