================================
Merge and Remove duplicated data
================================

This page shows how to check the timestamp of a new dataset, insert this data in the correct time series location in master file, and remove duplicated data.

Function of data insertion
--------------------------

.. literalinclude:: SRC/merge_2files.v02.py
   :language: python
   :lines: 104-109

.. literalinclude:: SRC/merge_2files.v02.py
   :language: python
   :lines: 120-130
   :emphasize-lines: 6

'df1' is the data of the new dataset, and 'dfm' is the data of the master file at each station.

Data insertion is performed when the station IDs (stid1, stidm) of the two files are the same.

Subroutine of data insertion
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This subroutine defiles the timestamps of the new dataset and master file and determines wherein the master file the new data will be inserted.

.. image:: SRC/merge_timeseries.png
   :width: 500
 
.. literalinclude:: SRC/merge_2files.v02.py
   :language: python
   :lines: 13-38
   :emphasize-lines: 2, 6, 9 


Sample program for inserting (time series) dataset into a master file
---------------------------------------------------------------------

This program gets a filename under multiple directories (station names) and reads the data in one file in each directory.

Source file: :download:`merge_2files.v02.py <SRC/merge_2files.v02.py>`