Efficient Timedelta Calculator

I have a time series data from a data logger that puts time stamps (in the form of dates MM--DD-YY HH:MM:SS:xxx:yyy (e.g. --[ 29.08.2018 16:26:31.406 ] --) where xxx and yyy are mi

Solution 1:

@zvone is correct here. pandas is your best friend for this. Here is some sample code that will hopefully get you on the right track. It assumes your data is in a CSV file with a header line like the one you show in your example. I wasn't sure whether you wanted to keep the time difference as a timedelta object (easy for doing further math with) or just simplify it to a float. I did both.

import pandas as pd

df = pd.read_csv("test.csv", parse_dates=[0])

# What are the data types after the initial import?print(f'{df.dtypes}\n\n')

# What are the contents of the data frame?print(f'{df}\n\n')

# Create a new column that strips away leading and trailing characters # that surround the data we wantdf['Clean Time Stamp'] = df['Time Stamp'].apply(lambda x: x[3:-4])

# Convert to a pandas Timestamp. Use infer_datetime_format for speed.df['Real Time Stamp'] = pd.to_datetime(df['Clean Time Stamp'], infer_datetime_format=True)

# Calculate time difference between successive rowsdf['Delta T'] = df['Real Time Stamp'].diff()

# Convert pandas timedelta to a floating point value in milliseconds.df['Delta T ms'] = df['Delta T'].dt.total_seconds() * 1000

print(f'{df.dtypes}\n\n')
print(df)

The output looks like this. Note that the printing of the dataframe is wrapping the columns around to another line - this is just an artifact of printing it.

TimeStampobjectLimitAint64ValueAfloat64LimitBint64ValueBfloat64dtype: objectTimeStampLimitAValueALimitBValueB0--[ 29.08.2018 16:23:41.052 ]--153.109302.9071--[ 29.08.2018 16:23:41.114 ]--153.020308.242TimeStampobjectLimitAint64ValueAfloat64LimitBint64ValueBfloat64CleanTimeStampobjectRealTimeStampdatetime64[ns]DeltaTtimedelta64[ns]DeltaTmsfloat64dtype: objectTimeStampLimitAValueALimitBValueB  \
0--[ 29.08.2018 16:23:41.052 ]--153.109302.9071--[ 29.08.2018 16:23:41.114 ]--153.020308.242CleanTimeStampRealTimeStampDeltaT  \
029.08.201816:23:41.0522018-08-2916:23:41.052NaT129.08.201816:23:41.1142018-08-2916:23:41.11400:00:00.062000DeltaTms0NaN162.0

If your files are large you may gain some efficiency by editing columns in place rather than creating new ones like I did.