Not properly accounting for market holydays (data shifts)

[pquan]

It looks like market holydays are not taken into account. For example, Thanksgiving 2024 was on Nov.28th, 2024. There was no market.
Downloading the daily NQ future from TV, I get a candle on 2024-11-28 and I'm missing the candle on 2024-11-29 which was the real open market day as follows:

                      symbol      open      high       low     close    volume
datetime
2024-11-04  CME_MINI:NQZ2024  20138.00  20248.50  20013.00  20086.00  520521.0
....
2024-11-26  CME_MINI:NQZ2024  20906.25  21024.75  20755.00  20993.50  505254.0
2024-11-27  CME_MINI:NQZ2024  20990.75  21019.00  20675.00  20813.00  520026.0
2024-11-28  CME_MINI:NQZ2024  20813.00  21017.50  20805.50  20993.50  339955.0

The data for 2024-11-28 should actually be the candle of 2024-11-29 (OHLCV values are correct for the 2024-11-29 date).

Further, looking at the TradingView Chart, i can confirm that the Candle for 2024-11-28 is missing, and the data is present only for 2024-11-27 and 2024-11-29.

My guess (to be verified) is that the same issue occurs on all market holidays and the data is shifted.

EDIT: Verifying the data that comes from TV, I see that the json value contains:

['i"', '134', '"v"', '', '1732748400.0', '20813.0', '21017.5', '20805.5', '20993.5', '339955.0', '}']

This means that, assuming the TV data is correct, some offset/timezone mangling is necessary to properly align the timestamp to a correct date.

A proposed fix:

Digging further, my guess is that:

• The timestamp returned by TV is localized in your particular timezone.
• The timestamp is for the beginning of the business day, and if the timestamp happens on a holiday, then it must be moved to the next business day.
  With these assumpions, he below proposed fix produces the correct timestamps:

from datetime import datetime, timedelta
import pytz

# Define the timestamp
timestamp = 1732748400  # 2024-11-28 00:00:00

tz = pytz.timezone("<your-local-timezone>")
dt = datetime.fromtimestamp(timestamp,tz).date()
print(f"Unadjusted date : {dt}")
if not is_business_day(dt):
    dt = next_business_day(dt)
print(f"UAdjusted date : {dt}")

the two functions is_business_day and next_business_day can be implemented using the pypi "holidays" package or the pandas_market_calendars package. your-local-timezone is the timezone of your TV account, or the default timezone if not using an account.

With this proposed fix, here's the corrected dates from the NQZ2024 contact:

                      symbol      open      high       low     close    volume adjusted_index
datetime
2024-05-27  CME_MINI:NQZ2024  19354.75  19450.75  19349.25  19420.75      74.0     2024-05-28
2024-06-19  CME_MINI:NQZ2024  20456.50  20601.00  20218.25  20286.00     728.0     2024-06-20
2024-07-04  CME_MINI:NQZ2024  20670.75  20893.75  20624.00  20875.75     817.0     2024-07-05
2024-09-02  CME_MINI:NQZ2024  19850.75  19919.25  19143.75  19234.50    4192.0     2024-09-03
2024-11-28  CME_MINI:NQZ2024  20813.00  21017.50  20805.50  20993.50  339955.0     2024-11-29

[pquan]

I'm still working on this issue, and unfortunately uncovered more timestamp troubles. Here's an example, using the same NQZ2024 contract. The timestamps returned by TV seem to not account for the specific day the day savings time start and end. Below is an extract from the returned data as a Timestamp and as a number. The data was downloaded from the Europe/Paris timezone (GMT+1 and GMT+2 when on day savings time).
The day savings end on Oct 27th, but the timestamp returned by TV seems to only change on Nov.1st. This results in a one-day jump and the weird timestamps on Oct 27 (sunday) at 23:00PM.

2024-10-23 00:00:00 1729641600000000000
2024-10-24 00:00:00 1729728000000000000
2024-10-25 00:00:00 1729814400000000000
2024-10-27 23:00:00 1730070000000000000
2024-10-28 23:00:00 1730156400000000000
2024-10-29 23:00:00 1730242800000000000
2024-10-30 23:00:00 1730329200000000000
2024-10-31 23:00:00 1730415600000000000
2024-11-04 00:00:00 1730678400000000000
2024-11-05 00:00:00 1730764800000000000

When changing the time zone from which you download data (like when using a VPN), the timestamps returned change as well, and they follow the local timezone from which you're downloading. However, the issue with the wrong day of DST end persists.

The problem seems to lie in the TV data itself and will require an ugly patch to fix, unless I am missing something, like perhaps a way to request timestamps explicitly in UTC.

I tried changing the wss message like this:

        self.__send_message("switch_timezone", [
                            self.chart_session, 'Europe/London'])

but the returned timestamps were identical. I'm out of ideas.

[ddouglas87]

This looks like it's an issue for specific symbols. E.g. testing it :SPX looks correct:

shape: (5_059, 8)

Date	Symbol	Exchange	Open	High	Low	Close	Volume
date	str	str	decimal[9,2]	decimal[9,2]	decimal[9,2]	decimal[9,2]	decimal[18,0]
2004-12-03	"SPX"	null	1191.35	1197.11	1187.70	1191.17	1566700000
2004-12-06	"SPX"	null	1190.84	1192.41	1185.18	1190.25	1354400000
2004-12-07	"SPX"	null	1190.44	1192.17	1177.07	1177.07	1533900000
2004-12-08	"SPX"	null	1177.07	1184.05	1176.80	1182.81	1525200000
2004-12-09	"SPX"	null	1181.18	1190.51	1173.76	1189.24	1624700000
…	…	…	…	…	…	…	…
2025-01-03	"SPX"	null	5891.07	5949.34	5888.66	5942.46	2113000000
2025-01-06	"SPX"	null	5982.81	6021.04	5960.01	5975.37	2678000000
2025-01-07	"SPX"	null	5993.26	6000.68	5890.68	5909.02	2612000000
2025-01-08	"SPX"	null	5910.66	5927.89	5874.78	5918.26	2480000000
2025-01-10	"SPX"	null	5890.35	5890.35	5807.78	5827.03	2956000000

The 9th was a trading holiday due to Jimmy Carter's death.

But FRED:BAMLH0A1HYBB not so much:

shape: (5_000, 8)

Date	Symbol	Exchange	Open	High	Low	Close	Volume
date	str	str	decimal[9,2]	decimal[9,2]	decimal[9,2]	decimal[9,2]	decimal[18,0]
2005-11-15	"BAMLH0A1HYBB"	"FRED"	2.89	2.89	2.89	2.89	0
2005-11-16	"BAMLH0A1HYBB"	"FRED"	2.96	2.96	2.96	2.96	0
2005-11-17	"BAMLH0A1HYBB"	"FRED"	2.86	2.86	2.86	2.86	0
2005-11-20	"BAMLH0A1HYBB"	"FRED"	2.88	2.88	2.88	2.88	0
2005-11-21	"BAMLH0A1HYBB"	"FRED"	2.90	2.90	2.90	2.90	0
…	…	…	…	…	…	…	…
2025-01-02	"BAMLH0A1HYBB"	"FRED"	1.76	1.76	1.76	1.76	0
2025-01-05	"BAMLH0A1HYBB"	"FRED"	1.71	1.71	1.71	1.71	0
2025-01-06	"BAMLH0A1HYBB"	"FRED"	1.75	1.75	1.75	1.75	0
2025-01-07	"BAMLH0A1HYBB"	"FRED"	1.79	1.79	1.79	1.79	0
2025-01-08	"BAMLH0A1HYBB"	"FRED"	1.77	1.77	1.77	1.77	0

You can see here the date is off multiple times. The 8th is supposed to be the 10th. It looks like multiple bugs at once.

What's happening here is TV is passing data in blindly from other sources. In this example FRED. FRED for some sort of reason missed the holiday and the data for the 10th says it's the 9th. (source) If I look at the TV webpage the the last date is the 9th, but if I have multiple data sources open into one plot the same data that said the 9th now says the 10th, which is correct. Somehow TV knows to fix the FRED data when multiple data sources are up on a single plot.

Then there is the bug in tvdatafeed that is listing the days one day off. The 5th should be Monday the 6th, and so on.

One theory I have is FRED:BAMLH0A1HYBB's smallest bar size is 1 day, where :SPX's is smaller than 1 day. That might have something to do with it. Otherwise I'm in the dark as to what the exact issue is.