Visualizing Tweets During Election Weekend

Posted on Thursday, Dec 10, 2015 in programming • Tagged with data, tweets, twitter, elections, visualize, folium, python, matplotlib, pylab

On Sunday (Dec. 06) parliamentary elections were held in Venzuela. I took the opportunity to fetch some data from Twitter's public stream and use graphics to show what was happening in "social media" during that day. To achieve this I relied in the Python programming language.

The Data

I decided to monitor all geolocalized tweets with my home city (Caracas) as origin over a period of five days, two days before the elections, the election day, and two days after it.

To fetch the data, I used the Twython library since I've worked with it before. Therefore, it should be easy to setup a quick script to fetch the data and save it in a text file.

Once the API and OAUTH keys are correctly following Twython's documentation, the next step is to fetch the data originating in certain location, passing it as a parameter to the streaming API. Twitter uses a set of bounding boxes to track the location of a tweet and only geolocated Tweets falling within the requested bounding boxes will be included.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
import json, os
from twython import TwythonStreamer

caracas_location = "-67.017225,10.419554,-66.778716,10.52006"

class MyStreamer(TwythonStreamer):
    def on_success(self, data):
        if data['coordinates'] is not None:
            with open('tweets.txt', 'a') as f:
                f.write(json.dumps(data))
                f.write('\n')

    def on_error(self, status_code, data):
        with open('errors.txt', 'a') as f:
            f.write('error: {0}: {1}'.format(status_code, data))



stream = MyStreamer(os.environ['APP_KEY'], os.environ['APP_SECRET'],
                        os.environ['OAUTH_TOKEN'], os.environ['OAUTH_TOKEN_SECRET'])

stream.statuses.filter(locations=caracas_location)

Since I just wanted to fetch geolocalized tweets, I checked for the existance of the coordinates entity in the …


Continue reading

Quick tip: Adding Variables to env in a Virtualenv (for development purpose)

Posted on Thursday, Jul 16, 2015 in programming • Tagged with python, enviroment, virtualenv, export, bash

If you're working with third party APIs, you might find code like YOUR_SECRET_KEY="some secret api key" in your source code, this is a bad practices for a lot of reasons (security, source code sharing, etc). Instead, the recommended way to manage this kind of situation is to add the value as a enviroment variable, and read it in your code with something like this:

1
2
import os
os.environ.get('YOUR_SECRET_KEY')

So, how do you avoid to add the variable to the enviroment each time you do some coding? If you're working with virtualenv you simply add it in the env/bin/activate script:

1
2
YOUR_SECRET_KEY="some secret api key"
export YOUR_SECRET_KEY

A Simple Django Project Layout

Posted on Monday, Mar 12, 2012 in programming • Tagged with django, documentation, layout, projects, python, settings, simple

UPDATE: This article was originally written when Django version was 1.4. Since this article was published I made changes based on the new features of Django. You can see some of them in this repository: https://github.com/jackboot7/django-project-template.

This is short entry on how I organize my Django projects, in a way that will be easy to deploy to a production server. This is not meant to be a full tutorial on Django, but just to document the way I layout my Django applications. I consider this layout a work in progress, as each day I learn something new that I can integrate into my project's layout. Anyway, I hope someone else can find this entry useful.

For most of my small/pet/side projects, I use a simple server schema, working only with a local development machine and a production server, I know that is a good practice to use a staging server with the same setup as the production server, but for small projects I found this unnecessary.

Project Layout

First, let me show you the layout of most of my projects:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
| - project_domain_name/
| | - .gitignore (.hgignore )
| | - requirements.pip
| | - env/                       # Virtual enviroment.
| | - var/
| | - server_config/             # Server configuration files.
| | | - nginx.conf
| | | - uwsgi.ini
| | |
| | - server_logs/               # Server logs.
| | | - errors.log
| | | - access.log
| | |
| | - project_dir/               # Django project. Most of the time I name it 'webapp'
| | | - apps/                    # Application apps.
| | | - templates/
| | | - static/
| | | - manage.py
| | | - urls.py
| | | - settings.py
| | | - local_settings.py

Ignored files

Some of this files and dirs are meant to be ignored by the control version system.

*local_settings.py*, for every local development machine, there should be a local settings file, this is where you put all the local development settings such as …


Continue reading

Python Audio Tools

Posted on Wednesday, Feb 15, 2012 in programming • Tagged with api, audio, audiotools, code, conversion, convert, encoding, example, flac, lame, m4a, mp3, programming, python

Some days ago, I started to work on a web application that requires to encode audio wave files (.wav) into three other formats, being: MP3, Flac and M4A. That encoding will run in its own Celery task.

One thing I knew for certain is that I wouldn't be running commands such as lame or flac directly.

Since the project is being written in Python, I though it would be nice to use a library to do all the encoding directly in Python code, without calling any external program..

If you're going to write an audio related application in Python, I recommend a visit to the audio section in the Python wiki, where you can find all the audio related modules that are available for Python.

I decided to use the API from Python Audio Tools, which is a very complete library to do audio manipulation. In the backend, it uses lame, flac and other commands to do the enconding of the files, but I won't have to run those commands directly.

So far, I'm very happy with the results I've got using audiotools, since it deals with metadata for the audio files (ID3v2 tagging, et al.) and it can even embed artwork into the files (if the format support such metadata).

I wrote a small Python script to show a basic use of this library, if you want, you can check it out on my githhub repo. I'll be adding a little more functionality to it as soon as I got some free time.


(Local) Online Python Documentation

Posted on Friday, Oct 15, 2010 in programming • Tagged with python, help, documentation

Today I learned about the pydoc command from Python. Even if you're offline you can access the standard library documentation running from a local copy using the command mentioned above.

Simply introduce this command in your terminal:

1
$ pydoc -p 8000

Then you can go to http://locahost:8000. This way you can browse the documentation for all your installed Python modules in the same way you'd use the 'help' command from the Python CLI.

[1] http://docs.python.org/library/pydoc.html

[2] http://pydoc.org/