Page 1 of 2 12 LastLast
Results 1 to 10 of 12
  1. #1
    Rookie Member NeuralTip's Avatar

    Converting news articles to numbers !!!

    Excited to join the community, I own my own site that has a following of around 50 people, now going live in a public beta.

    Anyways, I wrote up an article since I know a thing or two about machine learning/AI programming that might help a few people here and give them new ideas.

    Check out the article: http://neuraltip.com/turning-news-in...ated-approach/

    Basically, using a new method of unsupervised learning on sentence context of words, some Stanford guys figured out a method that gets pretty good results on flexible semantic representation of words.
    I've used it for a bit to turn my news into large vectors of thousands of dimensions. I correlate this to price movement along with other data usually used in Neural Network implementations in trading software.
    If you know a bit of Python or C you can easily use this and start converting your RSS news feeds into vectors to use as numerical indicators.

    Let me know what you guys think, IMO its a pretty untapped strategy especially if applied to StockTwit tweets (for high volume equities, lots of people telegraph their sentiment or bullish/bearish leaning).

  2. #2
    Master Member Bogdan G's Avatar
    You lost me there...

  3. #3
    Legendry Member Michael Hodges's Avatar
    Quote Originally Posted by Bogdan G View Post
    You lost me there...
    I think I get what the deal is here, however I'm not sure if this is an actual strategy or a new biz?

    YO, NeuralTip.... r u saying that u scan rss feeds for news trends and then somehow quantify that into a signal for trading?

  4. #4
    Rookie Member NeuralTip's Avatar
    Quote Originally Posted by Michael Hodges View Post
    I think I get what the deal is here, however I'm not sure if this is an actual strategy or a new biz?

    YO, NeuralTip.... r u saying that u scan rss feeds for news trends and then somehow quantify that into a signal for trading?
    Exactly, basically what happens is, training on tons and tons of sentences (first billion words from Wikipedia for example), a neural network learns to predict the words around a given word. The weights it gets to predict the surrounding words are used as descriptors for the given word. A vocabulary of all words the NN has seen is built, this vocabulary basically consists of for example:

    "NASDAQ" = vector <1.4, 2.6, 3.5, 0, 1.2, 2, 9.8, 6.6, 5.9, 1.1....> up to any arbitrary number of dimensions you select, these dimensions are the weights I mentioned earlier that result in correctly predicting the statistically probable words AROUND the word NASDAQ in the text corpus you trained on (financial news articles).

    So basically, you can turn any news article into an arbitrary array of vectors numerically describing the semantic concepts represented in that article. You can then trend/correlate these vector arrays with chronologically related price movements.

    For example, for News Article A about some U.S. regulatory move against banks that hits your news feed, you get N number of vectors from it and you see a corresponding non trivial price movement of X in direction Y for say 2 trading days after on some trading instrument you are tracking. You store that data.

    Now another article about similar legislation comes around again a few months down the road, but written differently and maybe containing new rules in the legislation. Having learned the concept vectors from last time, another learning model can learn that these X,Y and Z concept vectors make your instrument move a certain way. Thus, you can effectively create a signal. Of course, this learning process should be done across thousands of news articles.

  5. #5
    Legendry Member Michael Hodges's Avatar
    sounds pretty advanced. How far into the work are you guys?

  6. #6
    Rookie Member NeuralTip's Avatar
    Quote Originally Posted by Michael Hodges View Post
    sounds pretty advanced. How far into the work are you guys?
    Sounds advanced, but is pretty simple to implement, developing it for my own use and strategies and providing a service are two different things, I am working on it currently but getting access to news data that is consistent and garbage free (any news that is not company financials or economic related) is tough.

  7. #7
    Specialist Member LesterK's Avatar
    Very interested and very advanced! I hope I will understand it in details someday. For now I have only the basic picture. What kind of results did this scheme produced for you. Do you have any statistics?

  8. #8
    Rookie Member NeuralTip's Avatar
    Quote Originally Posted by LesterK View Post
    Very interested and very advanced! I hope I will understand it in details someday. For now I have only the basic picture. What kind of results did this scheme produced for you. Do you have any statistics?
    Well with companies that are popular to talk about online (Amazon for example) its pretty accurate, but I don't think I have a large enough news corpora to do any general commodity, sector or less known companies.

    From a day to day trade perspective it doesn't really help that much in terms of specifically predicting a target or direction, but I think where the real advantage it gives is adjusting your regular short term strategies for news. See most short term strategies attempt to chart out the battle of the speculators and short term emotions on the chart movement, volume, L2s and other highly granular data. Usually chart strategies involve waiting for movement based on fundamentals and news to die down and movement based on just trends of speculative supply and demand to take over which is much easier to predict and play with just charts and technicals.

    Converting news into vectors would allow you to arbitrarily adjust these strategies based on what news came out, effectively broadening the time frame in which you can apply your technicals strategy.

  9. #9
    Veteran Member hchandra's Avatar
    CMIIW
    This seems really advanced, I have heard neural network, genetic algorithm and fuzzy logic,
    converting RSS news to vector and use the value for calculation,
    there is one question popped into mind,
    This mean you must get news from respectable and trusted sources right?
    how if the source of news actually rewritten and posted in the website?
    around 80% of news circulating around the internet might be false news, how to filter it out?

  10. #10
    Rookie Member NeuralTip's Avatar
    Quote Originally Posted by hchandra View Post
    CMIIW
    This seems really advanced, I have heard neural network, genetic algorithm and fuzzy logic,
    converting RSS news to vector and use the value for calculation,
    there is one question popped into mind,
    This mean you must get news from respectable and trusted sources right?
    how if the source of news actually rewritten and posted in the website?
    around 80% of news circulating around the internet might be false news, how to filter it out?
    Restrict to only trusted news source (AP, Reuters, Yahoo Finance, Google Finance), and restrict to specific company symbol, so that company specific earnings reports, PR releases and announcements are most of the articles. Even then its tough, works better for short term with stocktwits as well.
    Last edited by NeuralTip; 11-26-2013 at 08:37 AM.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
3