Price History corruption/anomalies - cannot clean it up

jksatt's Avatar

jksatt

03 Feb, 2018 04:53 PM

Moneydance 2017.6 (1656) and Mac High Sierra 10.13.3

My problem is corrupted price history for every security (mostly mutual funds) I own. There are wildly incorrect values, e.g 15-20 times more than the correct price, interwoven among the correct history prices. Some of the price histories go back to the 80's because the transactions were originally imported into MD from Quicken. The dates with incorrect prices are not actual transaction dates. The prices on the transaction dates are ok.

What I've tried
1. manually using the "Remove" button in History to clear out a year or two of bad entries for a couple securities to see what happens.
The bad entries miraculously reappeared the next day.

2. Use the "Clear All" button to totally nuke one security's price history. Then use the "Import" button to add just the transaction history prices.
Seemed to work until the bad entries again reappeared in that security's price history the next day.

3. Use the "Clear All" button to wipe out the price history with no further action on my part. I.e., just leave the history section blank. The bad price entries and the good (transaction) entries again populated the price history without any invitation or action from me.

I cannot figure out what is causing the bad price history to keep coming back and attaching itself to its respective security/mutual fund.
And I cannot figure out how to actually delete the bad prices.

In the spirit of trying anything, I just uninstalled the Security Price Entry extension, and I changed the Quotes and Exchange Rates Updater extension to "do not update" for the Currency Exchange Rates. The Security Price is set to update weekly though sometimes I "Update Now." I have no other extensions.

Any other ideas on how to get rid of the bad price histories and make them stay gone?

Thanks, Julie

  1. 1 Posted by sth on 03 Feb, 2018 09:34 PM

    sth's Avatar

    Are you importing your prices from "transactions"? If so you need to fix the transactions or they will reappear.

    Item 3 seems to imply not, but are you set to autoload security prices when the program starts up? Have you got a bad security symbol in the list?
    I have been able to delete the old prices and they don't come back until I re-download them or import from transactions.

    Possibly you have a bad currency exchange rate? Are some securities on foreign exchanges different from your base currency?

  2. 2 Posted by jksatt on 04 Feb, 2018 03:15 AM

    jksatt's Avatar

    sth,
    Thank you for your input and apologies in advance for the following detail dump. FYI, all my transactions and accounts are in USD. For a number of years I had the security prices update when I opened MD, but in the last year I started updating just a couple times a week when I chose to update the prices.

    I was intrigued by your comment about a bad currency exchange rate. By way of background, in Nov 2016, I was fiddling with the foreign currency section on the summary page and for no good reason tried to add Mexican pesos to the list of currencies displayed. MD froze solid and I had to force quit. When I opened MD again all my investment account values had jumped more than 2,000%. I posted a question about the problem, but got no replies. After a day or two, I restored from an earlier backup and all was fine until my current problems.

    Fast forward to January 2018 when I was preparing to update to MD 2017. I checked to make sure my .moneydance data file and my .moneydancearchive backups were sitting in my ~/Documents folder as I had selected in MD preferences. I also used the MD File menu to make an extra backup via Export Backup to a USB drive.

    Out of curiosity I also clicked on MD Help>Show Archive Folder and found in the ~/Library/Containers/com.infinitekind.MoneydanceOSX/Data/Library/Application Support/Moneydance/archive a folder labeled Backups that had some backups from April 2016 (from just before I upgraded to MD 2015) and a Moneydance data file with the .md file extension. I dragged the backups to the trash. I left the .md file alone. It’s still there. Says it was created 13 Apr 16 and it’s about 15MB.

    Next clicked on Help>Show Documents Folder and I found in the ~/Library/Containers/com.infinitekind.MoneydanceOSX/Data/Documents a data file with the .moneydance file extension. Says it was created 13 Apr 16, modified 26 Nov 16, and last opened 19 Jan 18. About 75MB.

    My current data file resides in ~/Documents and it’s 111MB, created 29 Nov 16. My .moneydancearchive backups reside in the same ~/Documents folder.

    I am a basically tech-illiterate MD user. I was (and still am) confused by this 75MB .moneydance data file and it may be the source of my current price history troubles. Prior to the MD 2017 upgrade I wondered what that file was so I opened it on 19 Jan and was dismayed to discover it had all the bad investment account values from my November 2016 glitch. I hadn't remembered about the Nov 2016 account value problem until I opened that file. Anyway, closed it and returned to my good data file, investment account values were all fine and I proceeded with the upgrade to MD 2017.

    Truth be told, I honestly do not know if the price histories were messed up at the time of upgrade to MD 2017. I neglected to investigate.

    Got my Alpha Vantage key and started downloading prices again. All security symbols are good. It was only recently when I was inputting some data that I discovered the price problem and started trying to fix it as I described in my original post.

    I probably imported lots of bad price history data when I opened that 75MB .moneydance file. What I don’t understand is why deleting it is not working. And one other fact, all the bad price entries are prior to 26 Nov 2016. Everything from 26 Nov 16 through today is good.

    I am inclined to drag that 75MB file to the trash. Uninstall the Quotes and Exchange Rate Updater and clear all price histories. If the price history stays blank for a few days, then try importing from transactions.

    I would be very grateful for your thoughts or advice.

    Thanks so much, Julie

  3. 3 Posted by sth on 04 Feb, 2018 03:44 PM

    sth's Avatar

    It shouldn't import price history from one data file to the next. The price history is inside the data files themselves and kept separate.

    What I would recommend is to compress (zip) that 75MB file that you suspect and save it somewhere. Once it is a compressed zip file and moved out of the Containers it should be invisible to money dance. Then go back and try clearing your price history.

    I also keep my data file and automatic backups in the Documents folder. I open MD by double clicking the data folder or through a dock icon linked to the data file rather than the application.

    Possibly someone from Infinite Kind can help diagnose this issue. I am just a long time user.

    -Scott

  4. 4 Posted by jksatt on 04 Feb, 2018 04:08 PM

    jksatt's Avatar

    Thanks so very much for the reply. I will zip the 75MB file and move it. I will then clear a couple of securities of their price histories and see if they stay blank for more than 24 hours. I’ll report back. I have been using Spotlight to open MD. I’ll start going directly to the data file. Julie

  5. 5 Posted by jksatt on 06 Feb, 2018 08:24 PM

    jksatt's Avatar

    Zipped and moved the 75MB file but it seems to have made no difference.

    I’ve since had no luck with various ploys trying to clear a security’s price history in order to get rid of bad price history data that exists prior to 26 Nov 2016. With a few squirrelly exceptions that I cannot replicate, the bad entries mostly keep reappearing when I open Moneydance again.

    By way of a limited illustration I am attaching a screenshot of the price history for a 529 plan that I own. Purchased on 12 Jun 2015 at $17.06/share. The 7 dates in the history (prior to my possible above described Nov 2016 glitch) are actual dates that I manually entered a price in the price history for the 529 plan. It has no stock ticker symbol so in order to track it, I put the prices in based on its quarterly statements.

    I removed the seven bad entries using the “remove” button. The only price entries left were the five accurate ones from 31 Dec 16 thru 31 Dec 17. Closed MD. When I opened MD a few hours later, the price history for the 529 plan had reverted to the exact same picture as above. It appears to be impossible to delete these bad entries. The why escapes me.

    This is the same sort of thing that has happened with the one security with a much longer history that I chose to mess around with. I remove one by one bad price entries and they eventually come back. I “clear all” and good history might reappear once, but then it reverts to just entirely bad history. The "clear all" tactic seems to have only permanently removed the good history from 26 Nov 2016 to date. Choosing to import transactions works just once and then it reverts again to bad data.

    And the disturbing downside of this is that the two accounts that have this one security that I’ve been playing around now show in net worth and portfolio reports as both being worth 20+ times more than they actually are because the good history data is now gone thanks to my "clear all." The Summary page has the right figures and the actual account registers are all correct. Nevertheless, it is becoming more unnerving the more I work on it.

    I downloaded from my Backblaze a .moneydancearchive backup file from early January when things presumably were still good and before I upgraded to MD 2017. Recognizing that I will lose a month’s worth of inputs, I wonder if I should try to restore from this January backup file? I also have Time Machine if that would be a better path to follow.

    Thoughts and advice gratefully welcomed.

    Many thanks,
    Julie

  6. Support Staff 6 Posted by Henry on 07 Feb, 2018 11:00 AM

    Henry's Avatar

    Hi Julie,

    I apologize for the late response and I'm sorry to see that you are facing issues with Moneydance.

    Do you have syncing enabled?
    The reason for he question is that syncing could restore your previous state if the sync connection is facing an issue.

    Would it be possible for you to share your errlog.txt file?
    If you're using Mac OS X, it should be located at one of these locations within your Home folder:

    ~/Library/Containers/com.infinitekind.MoneydanceOSX/Data/Library/Application Support/Moneydance/errlog.txt

    or:

    ~/Library/Application Support/Moneydance/errlog.txt

    If you have any trouble finding your Home Library folder, open Finder and select Go -> Go to Folder and and copy and paste in the following:

    ~/Library

    This should pull up your local Library folder.

    If you're using Windows, it should be located at:

    C:\Users\(yourusername)\.moneydance\errlog.txt

    The errlog will reveal if the issue is a version specific bug, an issue with the sync connection or an issue with backups.

    Henry
    Infinite Kind Support

  7. 7 Posted by jksatt on 07 Feb, 2018 11:40 AM

    jksatt's Avatar

    Hi Henry,

    No syncing at all. I use MD only on my MacBook Pro. Automatic backups saved locally upon quitting MD. Time Machine backups and online backups via Backblaze.

    Attached (I hope) is err log.txt file

    Not to hijack the diagnosis but I feel the problem stems from the destructive event I had in November 2016. I eventually restored from a locally stored backup file and all was fine until January 2018 when I was preparing to update from MD 2015 to MD 2017. I came across a MD data file - probably from the Nov 2016 destructive event - and wondering what it was, opened it. At that point I think I baked the former problem into my current MD. As I noted to sth, I did not realize I again had corrupted price histories when I upgraded to MD 2017. All possibly leading to my present problem with trying to clear out price history and it not working as I described in previous posts.

    Thank you for your help!
    Julie

  8. 8 Posted by paul.r.smith on 07 Feb, 2018 12:36 PM

    paul.r.smith's Avatar

    I've found that my securities prices were not automatically updating as they always have.

    On checking the extensions, I saw that the 'Quotes and Exchange Rates' extension needed updating, which I did. It appears that quotes are now supplied bt AlphaVantage, rather than the previous Yahoo Finance. I duly got the API key and ran a test updatein the new extension. All the securities came up as Code 403 Forbidden, with no updates.

    I also downloaded an extension callled 'Download Quotes and Exchange rates Now', which doesn't have a pop-up window to show what is going on, and seems to have no effect.
    Where do I go from here?

  9. Support Staff 9 Posted by Henry on 07 Feb, 2018 03:49 PM

    Henry's Avatar

    Thank you for the errlog.txt.
    The logs do indicate that there is an issue with a transaction from 2016
    tiksync/out/20161126102352_800.txn
    I'm wondering if we can try to temporarily remove the transaction file and test if the price history accepts you changes.

    To do this, close Moneydance and navigate to your file
    /Users/juliesatterfield1/Documents/moneydance save file 2.moneydance

    Right-click (CTRL + Click) on the moneydance save file 2.moneydance file and choose the 'Show Package Contents' option, this will display a new window with a folder called 'safe', open the folder, within it should be a folder called 'tiksync', open it and within that folder there should be a folder called 'out', this folder includes the transaction that is presenting the error, please move the 20161126102352_800.txn file to a safe location (documents or desktop).

    Once removed open Moneydance and attempt to change the price history.

    The logs also indicate that the quotes and exchange rate extension is slightly outdated.
    Please open Moneydance and go to Extensions > Manage Extensions and choose the 'Update All option'.
    Once updated, go to the Extension menu and open the 'Quotes and Exchange Rate' extension and set the 'Currency Exchange rates' and 'Security prices' to Alphavantage, then click on 'Set API key' and enter the api key that you received.

    Henry
    Infinite Kind Support

  10. 10 Posted by jksatt on 07 Feb, 2018 05:52 PM

    jksatt's Avatar

    Hi Henry,

    I removed to the desktop the identified transaction file. The 529 account (earlier screenshot in my thread) for which I have to manually input prices, was missing the 7 bad entries when I just now opened MD after removing the transaction file. The one security (PTTDX) that I had been working on, still showed screen loads of 100% bad prices for the period 26 Feb 09 to 25 Nov 16. I used the “clear all” button - no import from transactions yet - and then closed down MD.

    Will reopen in a bit and to see what has happened with PTTDX’s price history.

    With respect to the extensions, I had uninstalled everything a couple days ago when I realized I had problems and have not re-installed any. Shall I go ahead and re-add the Q&E, or wait to see how the price history matter evolves?

    Thank you so very much for the help
    Julie

  11. Support Staff 11 Posted by Henry on 08 Feb, 2018 08:36 PM

    Henry's Avatar

    Thank you for reporting back.
    Please export a backup (just in case) and then attempt to re-add the Q&E extension and let us know if PTTDX’s price history is restored to the previous state.

    Henry
    Infinite Kind Support

  12. 12 Posted by jksatt on 09 Feb, 2018 03:07 AM

    jksatt's Avatar

    Hi Henry,
    Much appreciate your reply. Midday 7 Feb I followed your instructions to remove the transaction file to the desktop. As noted in my previous post I used “clear all” on the history for PTTDX. As of 2100 on 7 Feb and after having opened and quit MD six times during the day, PTTDX’s history remained clear/blank.
    Early this AM, I re-installed the Q&E and quit MD. Upon re-opening MD, the Q&E ran and downloaded prices for 31 securities. For the securities that still have bad histories, it just good prices for the period mid-Sep 2017 to date.
    With respect to PTTDX, it populated the history with good daily prices from 14 Sep 17 - 6 Feb 18. Those prices have stayed stable throughout today. Because of the seeming success with PTTDX, I cleared the price history on three more securities that had mixed bad and good prices and then downloaded good prices via the Q&E one by one. I.e., I unchecked all the securities that I own in the Q&E and chose to download just one security at a time for the three new ones I cleared. They have remained stable for about 14 hours now. MD has been opened and closed four times.
    However, the news is not all good. A couple hours ago, I chose two more securities, cleared their history, and used the Q&E to repopulate with good prices - each separately as I had done with the previous securities.
    Discovered just now upon opening MD that the bad histories are back for the latest two that I tried to clear, albeit in slightly different form than before. The bad history for these two latest securities stops on 26 Nov 2016 and there are no price entries until Alpha Vantage picks up in mid-Sep 2017 with all good prices.
    I will clear these two again now; leave them blank; download no prices and see if they remain clear tomorrow morning. If so, I will then try to download good prices using the Q&E.
    I will also make an export backup per your advice. I’ll let you know.

    Many thanks again for the help.
    Julie

  13. 13 Posted by jksatt on 16 Feb, 2018 02:21 AM

    jksatt's Avatar

    Henry,
    I wanted to sort of close the loop and tell you what has transpired since my last post. I have steadily cleared the price histories of 74 securities, 31 of which are still actively owned and have ticker symbols in the Q&E. In almost all cases there were years and multiple screen loads of price history. The first time I would choose "Clear All” the price history would initially look completely blank. Upon opening MD several hours later anywhere from one to two screen loads of prices would be back. Often the prices were all bad (wildly inflated) or occasionally mixed with a few good prices. All these self-regenerated price histories stopped on or shortly before 26 Nov 2016 — the date of the .txn file that you had me remove. With one exception, when I cleared the price history the second time, it seems to have stayed clear. A day or two later I started slowly using the Q&E to download the last 4-5 months of prices, just a few securities at a time.

    A couple of odd things: one security (no longer owned) took about 4 attempts to get clear. The second and third time, just one or two prices kept leaking back into the history. It seems finally cleared now and stable. And there were a couple securities, both currently owned and in use, that had years’ worth of price history and_no_bad prices. Completely at odds with all the other securities and their histories. I cleared them anyway and it also took two tries to get them to blank. Separately I discovered a bank account balance off and one of my credit card balances off and I found transactions for both on 26 Nov 2016 that had duplicated and downloaded again. I deleted the duplications and so far they have not re-appeared.

    Because all seems to be on an even keel now, I am reluctant to try “Importing from Transactions” in order to repopulate the securities with their older history. I am not dependent on this information so I doubt I will miss all that data. Do you have a view or advice on importing from transactions given what has transpired with my situation?

    And with respect to the .txn file that you had me remove, should I drag it to the trash or keep it around for a while in a safe place?

    Many thanks again for your able assistance.

    Julie

  14. Support Staff 14 Posted by Henry on 16 Feb, 2018 01:43 PM

    Henry's Avatar

    Thank you for the detailed description of all of the steps that you took and I'm very sorry to see how many were needed in your case.

    I am still curious about why the transaction caused the issue in such a way.
    You can delete the .txn file now as the state was restored.

    One last request form my side is, would it be possible for you to send us a new errlog.txt, it's mostly to see if there's anything else that we still need to take care of and as well to see the difference after removing the .txn file?

    Henry
    Infinite Kind Support

  15. 15 Posted by jksatt on 16 Feb, 2018 01:51 PM

    jksatt's Avatar

    Thanks, Henry. Per request, errlog.txt attached

  16. 16 Posted by sth on 18 Feb, 2018 01:00 AM

    sth's Avatar

    That file shows that quotes for AAPL downloaded correctly for the last 6 months. Sometimes the quote server gets overloaded and does not respond.

    THE OTHER PROBLEM THAT NEEDS FIXING - MD asks for quotes more rapidly than the quote server at AV allows and thus gets cut off after some number of quotes.

  17. 17 Posted by jksatt on 18 Feb, 2018 02:17 AM

    jksatt's Avatar

    Scott,
    You are right about AAPL downloading correctly and the same is true of my other active securities. I have never had any problems downloading good quotes during the Yahoo Finance era or now with Alpha Vantage. Those years’ worth of data however went totally haywire when I opened that bad MD file I had mistakenly saved prior to upgrading last month. While there may be bigger problems lurking, what struck my unskilled eye was that I could not clear a history, and have it stay blank, they kept self-repopulating with even weirder prices, until Henry had me delete that .txn file from 26Nov16 — the date of my original apocalypse in MD. Even after removing the .txn file it still took two separate attempts for each security to clear its history back to blank where I could start downloading from Alpha Vantage again.
    I just now re-check marked all my active securities in the Q&E and all 31 downloaded with just one error which said no data available. I am still very reluctant to try re-creating older history by importing from transactions. Since I don’t really need the history, maybe better to leave what I have alone?
    I can post an updated errlog if that would be more thorough for your perusal or Henry’s use.
    Thank you again for your help with my particular problem and thanks also for all you do to help others here.
    Julie

  18. System closed this discussion on 20 May, 2018 02:20 AM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac