On April 21st and 22nd, the Make Munich took place. A fair for all DIY aficionados, hobbyists, hackers and makers. There were lots of companies showing off what cool and creative things you can do using 3D printers and laser cutters and other technologies. On many booths you could get your hands dirty and “make” something yourself. Besides all this mesmerizing stuff that immediately makes you recall your creative childhood days, also a hackathon took place focused around Chrome Packaged Apps and the Parrot AR Drone. Some of us formed a team and participated, because who doesn’t like to hack some drones? We ended up hacking together an app that lets you control the drone via tweets. Admittedly a very limited, but nonetheless quite fun way to interact with the quadrocopter. In the end, we got to take home some prizes!
STYLIGHT @ ChromeDrone Hackathon: Dimi, Sergii and Dom
For those of you who have never heard of Chrome Packaged Apps or the Parrot AR Drone before: Chrome Packaged Apps are written in HTML5, JavaScript and CSS, but have to look and feel of native apps, meaning they don’t live inside the browser and hence, lack the traditional chrome (omnibox, tab strip, etc.). They are more powerful than web apps in some aspects and aim to be less dependent on network access (offline first approach).
The Parrot AR Drone is a quadrocopter, controlled over WiFi, that features an 720p/30fps front camera, as well as a vertical QVGA/60fps camera. The later being used for ground speed measurement. It also has a 3-axis gyroscope, a 3-axis accelerometer and a 3-axis magnetometer, a pressure sensor and additional ultrasound sensors. You can check out the full spec at the official site. Parrot also provides an SDK, which enables you to write your own software for the drone.
A few months ago, Paul Lewis and Paul Kinlan from Google demonstrated how they managed to control the drone via a gamepad from within a Chrome Packaged App using the new networking APIs of Chrome, as well as the Gamepad APIs in HTML5.
There are three UDP sockets and a TCP socket you can connect to in order to send and receive data from the drone. The drone sends navigation data, such as status, position, speed, engine rotation speed and the likes to the client on UDP port 5554. The video stream is transmitted over TCP on port 5555. Control and configuration data is send to the drone via AT commands (text strings) on UDP port 5556. The commands need to be send on a regular basis. For smooth movements at least every 30ms. The drone considers the WiFi connection as lost, if there are no consecutive commands within less than 2 seconds. The string commands basically consist of a command type and some arguments. For instance the progressive commands for movement take the percentages of the maximum configured angles for roll, pitch, gaz, yaw. I won’t go into more details on the commands here though, you can read up on the exact syntax in the docs bundled with the Parrot SDK.
Lewis and Kinlan wrote their own middle-ware for the drone in Javascript. The API establishes the four socket connections and generates commands in the required format. It also takes care of keeping the connection alive and initializes the drone when a client connects. They were kind enough to make their code available on github as part of the Chrome App Samples. This has kicked off some ChromeDrone hackathons. In Vienna for example, just a few weeks ago, a few people started hacking up the drone to be controllable by motion and others are still working on getting the video feed working.
Since we had never before played with the Parrot AR Drone or with Chrome Packaged Apps, we figured to start out simple and get a hang of the Parrot SDK and Chrome APIs. We initially had lots of ideas that involved the drone’s video or photo capabilities, but since the video feed wasn’t working properly yet, we scratched those and focused on different interaction methods. As mentioned earlier, controlling the drone via a gamepad had already been implemented, so we came up with the idea to use Twitter to control the drone. Tweet2Drone was born. The idea is to subscribe to either a hash-tag or a Twitter handle, which is then used to search Twitter periodically for tweets. The retrieved tweets are then put in a queue (DRONE.TweetQueue) which is in turn also periodically processed. If the tweet contains a specific command, it is send to the drone, if not, it’s just dropped.
We based our app on the code made available on github and stripped out the gamepad support. We provide a single input field in the GUI that let’s the user input the hash-tag/handler which should be used to search Twitter for commands and store it using Chrome’s Storage API, so you don’t have to enter it every time the app is restarted. As a bonus, we use the sync feature, which means, that input field is synchronized across all Chrome browsers you are signed in. This sounds like a lot of work, but it is actually super easy, basically you just specify you want to use storage.sync instead of storage.local and Chrome does all the rest for you!
Tweet2Drone Settings
For better control, we added a ‘start’-button to the interface, which triggers the init command of the drone’s API, establishing the socket connections and doing some basic configuration. It also reset the queue in case it contains tweets from a previous run, and starts searching Twitter. The Twitter search is a basic ajax request to ‘http://search.twitter.com/search.json?q=’ with our configured hash-tag/handler as the search query. We then process each individual tweet returned and check if it was previously inserted in the queue. If yes, we disregard it. If not, we extract the command from the message and push a json object containing the actual tweet message, the command and some meta data onto the queue. Afterwards, we use setTimeout to call the search function again.
The queue in turn pops and processes items every 3 seconds and calls the DRONE.Translator. The translator checks if a known command is present and executes the associated DRONE.API call. We chose this architecture, since it will allow us to easily extend the project and support other input methods such as text or speech.
Tweet2Drone in Action
The user get’s visual feedback in the interface about all tweets that were processed, as well as an indicator if they contained a recognized command or not. In addition, the interface has global notifications for connection related messages, e.g. when the emergency stop button was pressed, which causes the drone to land immediately and closes all socket connections. We also augmented the API to support the official emergency stop command, but deemed it a bit to brutal, since it cuts power to all rotors, no matter how high the drone is currently flying, causing it to drop like a dead bird.
This little hack is far from finished, since we only recognize and execute a few commands clean. We still have some trouble with timing rotation right and actually moving the drone in any direction in a controlled manner, but we had less than 24h to hack this up and get acquainted with both Chrome Packaged Apps and the Parrot AR Drone. The code can be found on github at https://github.com/lc0/Tweet2Drone. Since we now have a drone to play around with it, we might put some more work into it and polish it up. Feel free to contribute to this little hack!
