_Update: YAHOO servers not responding when QT tries to download data

Home ยป Topics ยป QuantTrader Forum ยป _Update: YAHOO servers not responding when QT tries to download data
_Update: YAHOO servers not responding when QT tries to download data 2017-05-20T16:22:32+00:00
  • Author
    Posts
  • Vangelis
    Moderator
    Post count: 119

    At the moment (18/5/2017) we are not able to get pricing data from Yahoo servers. The Yahoo server gives back the message “Will be right back” since Monday.

    https://forums.yahoo.net/t5/Yahoo-Finance-help/bd-p/finhelp

    We are working on this problem and will probably add a second data provider in the future for the case Yahoo does not work.

    We apologize for the inconvenience!

  • Vangelis
    Moderator
    Post count: 119

    Update 20/5/2017: Yahoo has changed their data service. This has affected a lot of software that retrieve data from their servers, including QUANTtrader. We are working on a solution.

  • PeterV
    Post count: 0

    Incredible but Yahoo has retired its widely used ichart service.
    See here: https://yahoo.uservoice.com/forums/382977-finance/suggestions/19310356-https-ichart-finance-yahoo-com-table-csv-s-spy-a?page=1&per_page=20.
    and here:https://forums.yahoo.net/t5/Yahoo-Finance-help/ichart-stopped-working-2-days-ago/td-p/251515

    For a long time there were three ways to get EOD data out of Yahoo:
    1. Scrape from the a stocks’ HistoricalData HTML page which is still possible
    2. Press the download button on a stocks’ HistoricalData page which is still possible
    3. Use the ichart API, which has been stopped.

    These three methods resulted in different data sets. The first used to be great a long time ago but gradually contained more and more artifacts (non-corrected splits and no dividends added in the adjusted close column. You would assume that 2 and 3 would give identical csv files but that was not the case, neither were 1 and 2 identical.

    In fact, only the third method was top-class. I used this to download hundreds of stock, ETF and mutual fund symbols each day and only seldom found discrepancies with other reliable data sources (like those on seekingalpha). I checked on a regular basis that this API method gave total returns, so dividends were injected in the curve and that was the case until they closed it down three days ago.

    To see how bad the first two methods are, just got to symbol SPXL, that had a split on 1 May 2017 and see on the screen and in the downloaded file that the data has not been corrected.

    Yahoo/Verizon have no intention to re-institute the service, unlike the message they are now displaying. This is the end of a highly respected and heavily used Yahoo service that cannot be replaced easily by any free source that I know of.

  • Alberto Magnani
    Participant
    Post count: 1

    I think that Yahoo have now lost their reliability. Anyway we can have some others alternative with API service for free EOD data:

    https://www.cityfalcon.com/
    and
    https://api.tiingo.com/docs/tiingo/daily

  • Vangelis
    Moderator
    Post count: 119

    Hello Alberto,
    Thanks for the pointers. The tiingo API could be useful.

  • PeterV
    Post count: 0

    Hi Alberto,
    Just got the Tiingo API integrated with my code. This is a very young company so we have to see if it is stable enough as yet and if they have the server capacity to cope when millions will join. But their data quality is excellent. The JSON is more informative than the former Yahoo CSV and is in correspondence with data from SeekingAlpha for about 100 random US stock and ETF symbols that I have checked so far. The number of decimals in their corrected prices is much higher than the usual 2 from other sources. One limitation is a smaller coverage than Yahoo ichart. In particular I am missing European exchanges, stocks and ETFs. But I’ve got everything going again thanks to your link.
    Peter

  • Vangelis
    Moderator
    Post count: 119

    We are working towards an updated version of QT with an alternative data source by end of the week (Friday 5/26/2017).
    Thank you for your patience.

    • dfbrobst
      Participant
      Post count: 3

      Are we going to be able to use intra-day data for QT? I’ve been running QT 90 minutes before the close on the last day of the month.

    • Vangelis
      Moderator
      Post count: 119

      Eventually you should be able to do intraday as well. Once we get the daily working we should have an ETA for that as well.

  • Gordon Cooper
    Participant
    Post count: 12

    Could Google Finance data be used instead of icharts api?

    Multiple Stock Quote Downloader for Excel

    • Vangelis
      Moderator
      Post count: 119

      Frank tried Goodle finance first. The data is unreliable even for liquid ETFs like “TLT”. You get horizontal lines because of no data updates for days. Quandl is another obvious choice but their premium service is @50/month.
      Yahoo continues its service so you can get data. The problem is that Yahoo, without notice, changed the API AND transitioned to unadjusted data. We need to take dividends into account so unadjusted data will not work for either reporting or ranking. We are now looking into Tiingo.com which is a fairly new service.
      This has been a big problem for a lot of people and self directed investors:
      https://forums.yahoo.net/t5/Yahoo-Finance-help/Is-Yahoo-Finance-API-broken/m-p/251312#M3123

      Reading the forum you can see there is a wide open space for a data provider to enter get data from the exchanges and adjust it for a small fee ($5-$7/month). I hope someone jumps in. We would support them ๐Ÿ™‚

    • BL Smith
      Post count: 0

      From what I remember for Google data a few years ago, it was not split adjusted data.

  • Mohamed Yosry
    Participant
    Post count: 7

    Is it possible to add API for TD Ameritrade to use brokerage data instead?

  • Vangelis
    Moderator
    Post count: 119

    We have updated Quanttrader to version 3.16S. Please go to the update forum or download from the link below:

    Version 316S changes:

    โ€“ This version switches to a new financial data provider after the Yahoo finance servers stopped to work for over a week.
    The download time for all symbols is still quite slow (about 10 min), but we will make it faster next week.
    Make sure you delete all old csv files if you install this version in a existing folder.
    When you start QT with one of your QuantTrader.ini files, then a popup window will tell you that some symbols have not been found.
    Pls. check then all symbold and then delete them.
    These symbols are non US symbols and are not used for our strategies. At the moment we can only download US symbols.

    โ€“ New feature: QuantTrader now does an automatic version update if QT finds a newer version on our server.

    https://dl.dropboxusercontent.com/u/43364046/Logical-Invest%20QuantTrader%20versions/QuantTrader316S.zip

  • mitch
    Participant
    Post count: 7

    I appreciate your efforts to find a FREE data service to replace yahoo.

    However,for those of us with paid data subscriptions, it would be VERY nice to have a plugin for our feeds.

    I personally subscribe to Norgate (https://www.premiumdata.net), but would be open to paying for another reasonable data provider.

    DATA is the LIFEBLOOD of the program and I am becoming very concerned with the reliability of FREE data sources.

    I’d rather have the option to pay for reliable inputs.

    • Vangelis
      Moderator
      Post count: 119

      Hello Eli,
      Thanks for the feedback. As a former (and current) Amiboker user I am aware of Norgate. Actually it was the first option we tried when Yahoo dropped the ball. The problem is that the data is not adjusted for dividends, which makes things a bit more complicated for us. Nevertheless, I do believe we need a option where a subscriber can import their own data (whether it’s from Norgate or another csv source) to QT so I will fwd the request to the team.

    • mitch
      Participant
      Post count: 7

      As a former (and current) Amiboker user I am aware of Norgate. Actually it was the first option we tried when Yahoo dropped the ball. The problem is that the data is not adjusted for dividends

      Hi Vangelis,
      I forgot to mention Norgate provides dividend adjusted data in their “beta testing program” for a new platform called Norgate Data Updater (NDU) they plan on releasing commercially “somewhere around the middle of the year.”

      Obviously we’re getting much closer to that time frame.

      Here’s part of an email they sent subscribers at the beginning of 2017….

      The commercial release of our new data updating platform is planned for mid-2017, and as such we are now ending the alpha testing program for PDU and moving into the beta testing phase with the next version of the platform, “NDU”.

      An overview of the beta testing program, along with instructions on how to formally request participation (should you wish to be involved), can be found here: http://www.norgatedata.com/beta-testing-program

      You may wish to check it out.

      Thanks again for everything you’re doing!

    • Vangelis
      Moderator
      Post count: 119

      Thanks for the update Eli. The bad news is the beta is closed for new users, the good news is Norgate is getting closer to a commercial release. We will follow their progress.

  • Gordon Cooper
    Participant
    Post count: 12

    Hi Guys-

    Any timeframe on getting mutual fund data to update?

    Also, should Interactive Brokers and Barchart.com be considered as potential data suppliers?

    Gordon

    • Rishi_at_Tiingo
      Participant
      Post count: 2

      Hi All,

      Thank you for the wonderful shout-out and blog post.

      WRT to Mutual Funds, I’m on it. Got contracts with 2 vendors and negotiating with a 3rd. Once that happens, expect a whole nother level of service and data integrity (a blog post will come out with launch). Alongside this, expect the entire US market to be covered. By the end of the new upcoming data release, expect coverage of about 41,000 tickers in the US alone. Once that stabilizes, I will expand globally.

      Just bear with me as I’m getting all the contracts in gear and scaling this operation up. There’s been a flood of new users. Thank you again!

      Rishi

  • Vangelis
    Moderator
    Post count: 119

    Hello Rishi,
    Welcome, thanks for posting and for providing a solution in troubled times ๐Ÿ™‚
    We look forward to the new release!
    Best,
    Vangelis

    • Rishi_at_Tiingo
      Participant
      Post count: 2

      Thank you Vangelis, right now I’m loading in an additional 20k+ OTC stocks and am gearing up for the Mutual Fund release.

      Could we talk offline about collaborating? I’m looking to launch a massive data release for a marginal cost of $7/mo – to help maintain the data, bandwidth, and add additional sources. Let me know! Rishi@tiingo.com

      Getting back to it!
      Rishi

  • Ivan Fisher
    Participant
    Post count: 38

    HI,

    I’m a bit out of the loop as recently as I havent been using QT, but I just installed V3.22S and tried to enter non USA tickers, QT seems to go off to Google finance for these but doesn’t return any data even though I’m using the google ticker format ASX:xxx ( Australia).

    Can somebody advise where are we at as far as non USA tickers since the Yahoo service was changed ?

    thanks
    Ivan

    • Alexander Horn
      Keymaster
      Post count: 263

      Since we have @Rishi in the loop, let me bounce this directly to him ๐Ÿ™‚

  • Charlie Moore
    Participant
    Post count: 7

    Hi,
    I was trying to use the Intraday prices option but I can’t seem to see the prices updating. I was wondering if it has to do with the data provider. I have selected Use End Of Day Data and Intraday Prices, with Tiingo as the Historical Data Provider.

    With these 2 selections, I Show Stock Data for SPY. I was expecting to see intraday prices show up here.

    Am I using this correctly?

    • Frank Grossmann
      Moderator
      Post count: 131

      You only get intraday prices 15 minutes after the US stock market opened. This because of the 15 min delayed data. You will see that you use intraday data if the last date on your chart is the actual date.

    • Charlie Moore
      Participant
      Post count: 7

      OK, I must be doing something wrong if you are seeing updated prices. It is now 1pm ET on 7/14/2017 and I have waited 20 min since starting and I only see prices for 7/13/2017. As before, here are the steps I go through:
      1. Login
      2. Select ‘Use End of Day Data and Intraday Prices’, Tiingo as the Historical Data Provider, click Start.
      3. Because this was the first time today starting QT, there was historical data to update from the FTP server. I selected to update the historical data.
      4. Once QT starts, I wait 15 min.
      5. In the backtest window, I select BRS strategy.
      6. Set Show Stock Data to SPY.
      7. Click Show.

      The window that pops up only has data up to 7/13/2017, which shows as the first line.

      I am running version 4.01S. Let me know if there is more info I can provide. Thanks for your help.

    • Alexander Horn
      Keymaster
      Post count: 263

      Hi Charlie,

      please try version 500, which has further options. A fresh data download for the standard portfolio takes no more than 5 minutes.

Reply To: _Update: YAHOO servers not responding when QT tries to download data

You can use BBCodes to format your content.
Your account can't use Advanced BBCodes, they will be stripped before saving.

Your information: