################################
Reading time-series station data
################################

This page provides how to **read hourly time-series station data** and **create timestamps from the days of the year** using Python. 

The sample Python program for reading the hourly data is as follows. 

Setup the modules
~~~~~~~~~~~~~~~~~

Importing the Python module to access the script from another Python file or module.

.. literalinclude:: SRC/read_files.py
   :language: python
   :lines: 1-11

Read the CSV data
~~~~~~~~~~~~~~~~~

The first step is to read `the CSV data <file:///Users/megchika/Box/Study/UCC/sphinx/uccda.0.2/build/html/dataformat.html#the-data-example>`__.

.. literalinclude:: SRC/read_files.py
   :language: python
   :lines: 95-99

'indir' is the directory where the data are stored, and 'filename' is the file name.

The original dataset contains several types of missing values, such as NAN, "NAN", and """NAN""". Add two kinds of NAN and "NAN" to the missing values information in the read module. (Python seems to recognize "NAN" and """NAN""" as the same format).

Generate timestamps
~~~~~~~~~~~~~~~~~~~

Next, generating timestamps using the days of the year and hour information.

.. literalinclude:: SRC/read_files.py
   :language: python
   :lines: 111
   :emphasize-lines: 1

.. literalinclude:: SRC/read_files.py
   :language: python
   :lines: 65-69

dayofY_toTimeStamp is the subroutine to pick up the date and time (YY-MM-DD HH:MM:SS).

Create new data frame
~~~~~~~~~~~~~~~~~~~~~

Create a new empty data frame with the timestamp index, add some data columns from the original data, and concatenate other datasets along wth the columns. 

.. literalinclude:: SRC/read_files.py
   :language: python
   :lines: 113-126

Save the data
~~~~~~~~~~~~~

Save the data to a CSV file with adding required header information.

.. literalinclude:: SRC/read_files.py
   :language: python
   :lines: 135-144

'tmpdir' is the temporal directory, and 'newfile' is the final data.

Sample program for creating a new dated data set (YY-MM-DD HH:MM:SS)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This program is designed to read all files under multiple directories.

.. image:: SRC/folder_info1.png
   :width: 300

Source file: :download:`read_files.py <SRC/read_files.py>`