Time Series Anomaly Detection with LSTM Autoencoders using Keras & TensorFlow 2 in Python

Показать описание

Detect anomalies in S&P 500 daily closing price. Build LSTM Autoencoder Neural Net for anomaly detection using Keras and TensorFlow 2.

Рекомендации по теме

Комментарии

if anyone has a problem with plot statements at the end, it helped when I used:
scaler.inverse_transform(test[TIME_STEPS:].close.values.reshape(1, -1)).reshape(-1),

and
scaler.inverse_transform(anomalies.close.values.reshape(1, -1)).reshape(-1),

MichalMonday

This is gold. I'm doing something similar for work. Glad I discovered this channel. Subscribed! Looking forward to more content!

CodeEmporium

Doing my bachelor's thesis on this. Very helpful gaining overview over the topic, thank you!

franziskahuber

Great tutorial. Just wanted to point out that the problem at 24:50 is in the way that the mean absolute error is calculated: np.abs(X_train_pred, X_train) is no the same as np.abs(X_train_pred - X_train)

leoparada

Dear Venelin,

You are training your model using labels (y_train) which are t+1 timestamps for each training sequence (X_train), but Autoencoder is implied to train model with labels that is the same as input training sequence, it means you should use model.fit(X_train, X_train), I guess. Could you kindly explain why you use this scheme of training and name it like Autoencoder ?

mikhailb

Appreciate all your help man, it's really made a difference in how quickly I've learned a lot of these concepts! You are the best.

christalone

but why is the test loss constantly increasing doesnt that mean the model is not learning well so the anomalies can be wrong ?

SurajBotcha

This is exactly what i wanted to learn.. would you be able to do the same for a multistep multivariate time-series and identify the anomaly and forecast? Thanks!

vigneshpadmanabhan

Hi, why do we use y_train to fit the model and not X_train? it is autoencoder right? so we train the model to be able to reproduce the input so X_train

martintabikh

Really really helpful to help me with my time series problem related to climate change, thank you!

shyamkarthikrameshbabumis

Great work Venelin. Very Clean step by step explanation. Keep it up

pratiksingh

Thank you for the tutorial. There is one piece I didn't understand. The shape of y_test is 380, so I was thinking that the model would make 380 predictions, but the shape of the predictions (y_test_pred) is 380*30. Is it making the 30 predictions per date? For example it uses the prior 30 days as the input sequence and its predicting the next 30 days? I was also thinking since the shape of y_train is a single closing price per day that the model would be trained to only predict one value per date, not 30. Can you clarify?

donwoodlock

Great Tutorial, it's really noticeable that you know what you're doing. Keep it up

xRandom

Excellent video, thanks a lot! However I would like to see an extension of this example using multiple features. I tried to extend it using more features with no luck so far.

xenophon

I am wondering a little bit what we gain from detecting historical anomalies?
It is like knowing that last weeks weather probably had an anomalie.
What we need is a prediction (not of the future price, that is not enough) of the future performance.
If we can't have this than we would at least like to have a prediction of future anomalies.

hipphipphurra

Hey Mr. Venelin, thank you for the video. If you allow me to ask you some questions, why do we have, while train the model, pass the X and the Y? Is the model reconstructing the original sequence and trying to predict the next value based on the 30 values provided? (I am asking because I was expecting that we would bass the same sequence, something similar as we perform using a vanilla autoencoder). It seems that we input a sequence, tries to predict the next value for the given sequence while we reconstruct the initial sequence.

When we calculate the error, the error is based on the reconstruction process am I right?

Thank you in advance!

Breno

At around 24:05 time, it should be np.abs(x - y) and not np.abs(x, y) right?

rhithickm

Thanks for the tutorial. Question: Why do you create target label (y), when you are doing an unsupervised analysis?

maziarkasaeiroodsari

I have an ad every 3 minuts... Except that, excellent video. However the anomaly detected don't seem to be abnormal for me. It' more abnormal when the change is big and sudden

FRUXT

Hi Great tutorial. Trying to utilize your code with some of my data. Only change was number of time steps. Getting shape incompatibility errors when I try to use y_train in the fit method. Also using Keras 2.2.4 and tensorflow 1.13.1 ... Any advice?

DanBarbatti

Time Series Anomaly Detection with LSTM Autoencoders using Keras & TensorFlow 2 in Python

Anomaly detection in time series with Python | Data Science with Marco

Anomaly Detection : Time Series Talk

Time Series Anomaly Detection Techniques for Predictive Maintenance

Anomaly Detection For Time Series Data in Python

Shreya Khurana - Realtime Time Series Anomaly Detection in Production | PyData Global 2024

Time Series Anomaly Detection Using Prophet in Python | Machine Learning

New Trends in Time Series Anomaly Detection

Time Series Anomaly Detection with Deep Learning - Sergei Bobrovskyi, Airbus

Anomaly Detection with Time Series Forecasting | XenonStack

Anomaly Detection using Isolation Forest - Time Series

How ANOMALY DETECTION works in time series using the Holt-Winters Algorithm

USENIX ATC '21 - Jump-Starting Multivariate Time Series Anomaly Detection for Online Service Sy...

Time Series Anomaly Detection Tutorial with PyTorch in Python | LSTM Autoencoder for ECG Data

ARIMA and time-series anomaly detection with BigQuery (The Data Show, w/ Felipe Hoffa & Yufeng G...

Time Series Anomaly Detection with Machine Learning - Sergei Bobrovskyi, Airbus

Isolation Forest for detecting anomalies in time series

Time series anomaly detection in real time.

180 - LSTM Autoencoder for anomaly detection

Anomaly Detection in Time Series Data with Python

Kiraly, Risi, & Tveten - sktime: time series anomaly detection, changepoint detection, segmentat...

Time Series Anomaly Detection with LSTM Autoencoders using Keras & TensorFlow 2 in Python

Timeseries anomaly detection without anomalies

Anomaly detection in time series data

Time Series Anomaly Detection in Plaintext Using Apache Spark with Jerry Schirmer (SparkCognition)