################################
Reading time-series station data
################################

This page provides how to **read hourly time-series station data** and **create timestamps from the days of the year** using Python. 

The sample Python program for reading the hourly data is as follows. 

Setup the modules
~~~~~~~~~~~~~~~~~

Importing the Python module to access the script from another Python file or module.

.. literalinclude:: programs/read_merge_files.py
   :language: python
   :lines: 1-12

Read the CSV data
~~~~~~~~~~~~~~~~~

The first step is to read `the CSV data <dataformat.html#the-data-example>`__.

.. literalinclude:: programs/read_merge_files.py
   :language: python
   :lines: 157-159

'findir' is the directory name where the data are stored, and 'filename' is the file name.

The data contains four header rows. Therefore, skip three rows and use one header row as headers (column names).

The original dataset contains several types of missing values, such as NAN, "NAN", and """NAN""". Add two kinds of NAN and "NAN" to the missing values information in the read module. (Python seems to recognize "NAN" and """NAN""" as the same format).

Generate timestamps (YY-MM-DD HH:MM:SS)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Next, generating timestamps using the days of the year and hour information.

.. literalinclude:: programs/read_merge_files.py
   :language: python
   :lines: 171-174
   :emphasize-lines: 4

.. literalinclude:: programs/read_merge_files.py
   :language: python
   :lines: 115-119

dayofY_toTimeStamp is the subroutine to pick up the date and time (YY-MM-DD HH:MM:SS).

Create new data frame
~~~~~~~~~~~~~~~~~~~~~

Create a new empty data frame with the timestamp index, add some data columns from the original data, and concatenate other datasets along wth the columns. 

.. literalinclude:: programs/read_merge_files.py
   :language: python
   :lines: 176-189

Save the data
~~~~~~~~~~~~~

Save the data to a CSV file with adding required header information.

.. literalinclude:: programs/read_merge_files.py
   :language: python
   :lines: 205-211

.. literalinclude:: programs/read_merge_files.py
   :language: python
   :lines: 82-114

'newfile' is the final data.

Organize file and directory structure
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This program raads all files under multiple directories, merges multiple files into one file per station each year, and saves the files.

Initial directory structure:

.. image:: images/folder_info.ver02.01.png
   :width: 300

Output directory structure:

.. image:: images/folder_info.ver02.02.png
   :width: 300

Source file: :download:`read_merge_files.py <programs/read_merge_files.py>`