Natural Language and Market News

Hello my friends, today I am presenting a very short article on how natural language and web scrapping can help us to stay tuned with market news and trend.

It’s not news for any of you that I am web scrapping news website for a long time since 2012/2013 my algos are collecting news from different sources on the internet. This week I decided to show you how I use this in my favor and some of the coded used to do it. 

First thing I did was prepare my database to work with FULLTEXT search, I did this with the following command:

ALTER TABLE noticias ADD FULLTEXT(data, resumo);

The table name is “noticias”, noticias é news in Portuguese. What I did use to create a special kind of index that makes text search queries faster.

So I defined extra two functions to search in the database, I will not display the MySQL getQuery function, it’s just a wrapper to MySQL so nothing interesting to see.

This function searches for a “term”, some “lookback” “timeframe” ago and the timeframe can be [hour, day, month, year] and will return a list of links ordered by date.

Check this:

def pesquisarnoticia(term,lookback=1,field="link",timeframe="day"):
    query ="select {} from noticias where  data > now() - interval  {}  {} and match(titulo, resumo)  against('{}' IN BOOLEAN MODE) order by data"
    busca = query.format(field,daysago,timeframe,termo)
    return [str(x[0]) for x in getQuery(busca,'forex')]

This other function will first use the “paresforex()” function to get all the available symbols I track in my database and then search for news that contains the symbol in the title or in the news body. 

def morningnewswords(field="link"):
    termos = paresforex()
    noticias = {}
    for termo in termos:
        noticia = pesquisarnoticia(termo,, field, "hour")
        if len(noticia) > 0:
            noticias.update({termo: wordsfreq(1,Noticia = noticia)})
    return noticias

The function returns a dictionary of news as values and symbols as keys. Check the results I got running it now! 

a = morningnews()
{u'$aapl': ['',
 u'$qqq': ['',
 u'$spy': ['',
 u'audnzd': [''],
 u'audusd': ['',
 u'dxy': [''],
 u'eurchf': [''],
 u'eurusd': ['',
 u'gbpusd': ['',
 u'usdcad': ['']}

Remember our trade of the week on Orange Juice?  This is how I found the articles related to it:

noticia = pesquisarnoticia('"ORANGE JUICE"',field="link",daysago=30)

This function is a little bit more interesting, it basically counts which words appear the most with the term I searched. Let’s try with “dollar”:

def wordsfreq(termo,lookback=1,termo2=None,noticia=None):
    if noticia is None:
        noticia = pesquisarnoticia(termo, lookback)
    noticiaunica = ' '.join([noticia[0].lower() for noticia in noticia])
    sentences = stok.sentences_from_text(noticiaunica)
    if termo2 is not None:
        sentences = [sen for sen in sentences if termo2 in sen]
    d = []
    for sent in sentences:
        for w in word_tokenize(sent):
            if w not in stopwords.words('english') and len(w)>2:
    bot = nltk.FreqDist(d)
    both_most_common = bot.most_common()
    d = list(itertools.chain(*(sorted(ys) for k, ys in itertools.groupby(both_most_common, key=lambda t: t[1]))))
    return d

Let’s run and see the 20 words that appear the most in the news that contains the word dollar


words = wordsfreq("dollar")
[(u'trade', 201),
 (u'trump', 190),
 (u'said', 189),
 (u'market', 173),
 (u'dollar', 152),
 (u'president', 144),
 (u'u.s.', 141),
 (u'gold', 122),
 (u'bank', 112),
 (u'china', 106),
 (u'impeachment', 106),
 (u'would', 100),
 (u'fell', 97),
 (u'2019', 93),
 (u'company', 92),
 (u'stocks', 91),
 (u'tuesday', 87),
 (u'markets', 84),
 (u'per', 84),
 (u'also', 83)]

That’s it, my friends, this week I am launching a functionality inside the chatroom where users will be able to conduct the news of search with a command like 

/newssearch "words I want"

When I say coding is powerful is no joke at all this can leverage your trading a lot, if you need assistance coding with a trading idea you have, just leave a reply that I will contact you.

I hope you all liked it and if you have suggestions about articles please leave a comment.

My best regards

Leo Hermoso


Comments (2)

Mate i just wanted to say thank you, i’m getting passioned about python and coding thanks to you, being able to analyze this type of data can be very helpful in trading, please keep posting content, i’m loving it, will soon join your chat too

Hey bro! Nice to hear that! Check this link you will like it

Leave a comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.