Generic Speech Recognition Script for SL4A

I've finally had some time to make a generic speech recognition script that hopefully any SL4A capable Android device can use. I've taken parts of my script from the previous post and added in some sample pattern matching from my server side script. The result is a script that can issue commands to your home automation controller or software by fetching URLs. A couple prerequisites: you need you must have SL4A and Python for Android installed on your device. It would help to be familiar with some Python and its regular expression syntax. The code is well commented and has samples for recognizing phrases like "turn off the kitchen light" and "turn the master lights off" - so hopefully that's enough to kickstart your automating. So go ahead and get it!

More Two Way Interaction With Android Speech Recognition

Let's start with the demo first. The video shows commands and queries being spoken and recognized by Android speech recognition. Our web GUI is also in the shot - since the wife forbids me to walk around filming the insides of our house for the world to see :) - so you can at least "see" some of the status being queried and the results of some actions. There are some annotations on the video, but you can't see them on the embedded player. Click through to YouTube to see the video with annotations.



What makes all that stuff work is the queries are passed to a server for processing. That gives the opportunity for two way interactions where you can not only control your system but query it as well. As I mentioned in my previous post on this topic, using IM as a transport mechanism allows the recognized phrase to be sent to the server and the responses sent back to the Android device. Over on our server, EVERY device and its state is logged in our MySQL database. This was done when we built our AJAX based GUI. Also, since our system is distributed, MySQL provides a place for status to be updated and synced between various devices. Below is a snapshot of a phpMyAdmin page showing part of one of our database tables. The table contains the device name, type, its state and when it was turned on and off.



Every device and its state is stored: every light, appliance, AV device, motion sensor, door, window, lock, car, phone, computer, etc. Whenever a device's state changes, a function gets triggered in whatever software is interfacing to that device (which is for the most part Windows scripting like the jScript below):

function setStatusOnOff(device,type,state,secs) {
    try {
        if (state=="off") {
            mysqlrs.Open("insert into status (device,type,state,secs_off) values ('"+device+"','"+type+"','"+state+"','"+secs+"') on duplicate key update state='"+state+"', secs_off='"+secs+"'",mysql);
        } else {
            mysqlrs.Open("insert into status (device,type,state,secs) values ('"+device+"','"+type+"','"+state+"','"+secs+"') on duplicate key update state='"+state+"', secs='"+secs+"'",mysql);
        }
...
}

Since the device name is stored as a normal non-abbreviated name ("family room tv" instead of "frtv"), it's straightforward to use the recognized speech to search for devices using MySQL queries. The next step is to figure out what type of command is being issued. For example, a command will have the phrase "turn on" or "turn off" in it. Since I use Python on the server to process the speech, I use its regular expression (regex) functions to pattern match for commands:

reTurn = re.compile('(^|\s+)turn.*\s+o(n|(f[f]*))($|\s+)',re.I) # recognize "turn on", "turn this and that on", "turn this and that blah blah blah off" or just "turn off", even "turn of" anywhere in a sentence

After figuring out if it's a command or query, my script then strips out extraneous text to simplify extracting the device and type. What gets stripped out depends on how things are phrased in your household. Here's a snippet to do that, where msg is the recognized phrase:

msg = re.sub('(^|\s)+(turn)|(the)|(a)|(can)|(you)|(please)|(will)\s+',' ',msg) # strip out unneeded text

I'm experimenting with natural language processing to strip out unnecessary words automatically but it's not ready yet. Next, the script figures out the type of device involved. For lighting, it would use a regex similar to this:

reLight = re.compile('\s+(light[s]*)|(lamp[s]*)|(chandalier[s]*)|(halogen[s]*)|(sconce[s]*)($|\s?)',re.I)

Since all the extra words have been stripped out and the type has been determined, all that's left is to formulate a MySQL query like this to get the actual device name:

msg = re.sub("\s+","%",msg) # replace spaces with wildcard character %
if reLight.search(msg):
  query="select * from lighting where device like '%"+msg+"%'"

This is necessary to remove ambiguities in the recognition. A light may be named "guestbath" in the HA system, but Google may pass the recognized phrase as "guest bath." With the actual device name, the final steps are to issue the command and send a response back to the Android device. As lights and other devices are added to HA system, nothing else needs to be added. Contrast that with other automation systems where you have to setup a recognition phrase for every device and possibly every state in your system. In our system, new device names will be parsed out of the database, and no changes are required on the Android device. Queries also follow a similar flow, except instead of issuing a command, a response is formulated with the status and sent back to the user.

That's the backend. I'll cover the frontend in another post.

Using CanvasJS to Graph Power Consumption

We've been using RRDtool for graphing everything from temperatures, to disk usage, to power consumption. It's very powerful and makes some nice charts, but I can never remember how to set up the database. Plus, my server is constantly running the tool to generate the graphs every 15 minutes so it's relatively up to date when someone views them. I'm now playing with CanvasJS which uses HTML5 and JavaScript to easily generate some really cool graphs. I'm using power consumption as my test bed for implementing CanvasJS. Data for the power consumption is dumped into our MySQL database every 2 minutes (it's actually coming in every second, but I'm only sampling the data every 2 minutes for this graphing application). With some JavaScript and PHP pulling the data out of MySQL, the charts are generated on the fly. It works REALLY well and it's fast. You can pan and zoom the chart to see the exact power usage at a specific time. Check out the gallery for more samples with code. I will probably transition all our system's graphing over to CanvasJS, after I have more time to experiment. In the meantime, here's a short video showing the power consumption graphs I'm working with.

Flat UI Update to GUI

Our major HA user interfaces are web based, and when I see prepackaged UI elements, I think about how I can use them to improve what I have. We mainly use our custom floorplan GUI, but occasionally use some older interfaces. One of them is below, design circa 1999 :) As you can see, the artistic side of my brain isn't very developed.



It's actually the 2nd generation - the first was ASP and IIS based. The 2nd version didn't change the UI at all, just the implementation, which I switched to Perl CGI and Apache. I ran across a Flat UI package some time ago. Flat, from what I've gathered, is how many of the newer, hip websites are designed. I figured I'd give it a try and bring some of its design elements into our UI. As a tutorial, I've redone the above lighting UI with toggles and sliders from Flat UI. You can see how much nicer it looks in this short clip:



It's also more finger friendly for tablet use. I made my own tweaks to fit in my existing template, but I still need to move things around a litle (slide the toggles down a few pixels, maybe reduce the open space). It was relatively painless (I already know enough JavaScript and JQuery to figure out most of the kit), and I can see using it in other interfaces.

Using SL4A and Android Speech Recognition for Home Automation

My latest project has been experimenting with SL4A (Scripting Language For Android) and Python on my Galaxy Note II. I started with the included saychat.py sample to build a simple script that kicks off Android speech recognition. It takes the text result returned from Google and sends it over IM to our HA server. The HA server does some basic natural language processing on the text, extracting commands and performing the operations if any valid ones are found. It then returns a response over IM to the phone with the result of the command(s). Back on the phone, the Python script has been waiting for this confirmation and uses TTS to read it back. The cycle repeats until the user says "goodbye" or it gets two consecutive recognition results with no speaker. Here's a short YouTube video of it in action:



From the video, you can see I've tried to parse the speech so that it can find the commands and devices even if the command is spoken differently. I used three different phrases:

"Can you turn on the kitchen light and dining room light?"

"Can you turn off the lights in the kitchen?"

"Turn off the dining room light."

I was trying to avoid having only simplistic commands like the last one. The first one demonstrates that ability to speak a command for multiple devices, and the ability to preface the command with "Can you" or pretty much anything like "The dog wants you to" ;) The 2nd command shows that it's not restricted to parsing "kitchen light" together. The last command is a typical HA VR command. My parser also has the ability to decode multiple commands in one, such as "Turn off the kitchen light, the guest bath fan and living room light and turn on the back floods." The only challenge is saying everything you want to say without much of a pause, otherwise recognition stops and the partial command will be sent.

A few advantages of using this setup:

Google's speech recognition in the cloud is probably the best, most up-to-date system. They started building up their system with the now closed GOOG-411 service. Further fine tuning gets done on the millions of voicemails their Google Voice service transcribes. Their Chrome browser also uses their speech recognition and of course, so do the millions of Android users. All this input goes into tuning their accuracy, and what you end up with is one of the best performing, up to date speech recognition systems. If you're using Microsoft's Windows VR, you're probably getting something that gets updated every few years with each OS release - if you're upgrading. With HAL, you've getting a 1990s VR engine. I'm not even sure if that gets updated anymore.

Google's free form speech recognition allows the most flexibility in speaking commands. Granted, that makes the parsing more difficult, but it allows a system that can more accurately respond to the different ways different people phrase commands. Most speech recognition engines I've worked with require you to pre-program canned phrases in order to recognize commands. If you deviate just a little from what's programmed, good luck getting your command recognized.

By using Jabber IM as a transport mechanism for the recognized commands, the same system that works at home, works when you're away. You just turn on your mobile data - there's no VPN or SSH tunnels to set up every time you want to speak a command. There's one level of security for free since your home's IM client must have pre-approved other users to allow communication (adding them to the roster). Another level can be done at the scripting layer of your HA software, by limiting what IM users can issue certain commands. For extra security, you can even encode or encrypt the text being sent over IM if you want, but if you're using Google Talk servers, your communication is already wrapped in SSL.

A few more details. Using SL4A, I cannot control the default speech recognition sounds - it can get annoying after a while. I'm using Nova Launcher as my launcher instead of TouchWiz. Nova Launcher let's you remap the home key behavior on the home screen. When pressed, instead of showing the zoomed out view of all my screens, it kicks off the script. Also, my HA device database is stored in mySQL, which allows for powerful searches and easy matching of what's spoken to actual devices - even when the device name isn't exactly the same as what was spoken. I've been using the mySQL setup, IM interface and command parsing for many years now, (although the parsing was more primitive) so integration was extremely simple. At some point, I would like to implement NLTK, the Natural Language ToolKit, for more complex language processing.

2012 Most Downloaded (3 months late)

Here is our list of popular downloads for 2012:

1. EventGhost xPL Plugin - 99 times
2. xScript - 77
3. BlueTracker - 61
4. xPLGVoice - 39
5. xPLSerial - 17
5. BlueTrackerScript - 17
7. xPLGCal - 16
8. t2mp3 - 15
8. Blabber - 15
8. Noise - 15
8. xPLChumby - 15

The EG plugin continues to be popular despite our stopping development on it years ago. Four of the top five and their downloads are almost exactly the same as last year, with xPLChumby dropping and xPLGVoice getting more interest. There are no new apps on the list since I did virtually no HA last year, but it's nice to see there's still a similar amount of interest in our existing apps. I haven't had the urge to code anything new, let alone time to brainstorm new things. We'll see what 2013 brings...at least I'm blogging again.

Solar Year #3

We just finished our 3rd year of having solar panels, where we generated nearly 75% of electricity usage and saved $1107. After the first year, we've done a good job of reducing our electricity usage by unplugging unnecessary devices and consolidating servers. However, I'm always looking to buy more gadgets :).

year	total usage kWH	solar kWH	grid kWH	savings $	avg monthly kWH	% solar usage
1	8636.35	5811.2	2825.15	$1,234	719.7	67.3%
2	7640.2	5739.2	1901	$1,144	636.7	75.1%
3	7410.95	5548.95	1862	$1,107	617.6	74.9%

Total output from the panels has decreased each year, but that's been mainly due to a larger number of rainy or overcast days. I'm sure some of it is due to dust collecting on the panels. I have yet to get up on the roof to wash them off, and it's something I plan on doing this spring. The wet winter we've had this year has done a decent job of washing the panels for me. Overall, we're very pleased with our solar panels, which have already paid for 28% of the cost (slightly less than I projected due to us using less electricity).

Tasker $1.99 at Google Play

Tasker, an app for automating almost anything on your Android device, is on sale at the moment for $1.99 (regularly $6.49). I was interested in buying it a while ago, but $6.49 for an app seemed a bit high. Glad I waited. Get your copy at Google Play.

Automated Barking Dog Correction

Our Australian Shepherd is a wonderful dog, but he's a bit protective of our property. When he's out in the backyard, he will bark when he senses anything unfamiliar to him in front of our house. He tends to bark near the gate, so I've wired a microphone placed in a garage vent near the gate back to a server. On that server, we've been using our Noise app to detect when he's barking. It generates an xPL message when the sound levels detected exceed a specified threshold. There's a few conditions that are checked before the system decides if he's barking, like if the gate and garage door are closed and there's motion detected near the gate. The server will then play recorded MP3s of our voices telling him to be quiet through a speaker placed near the same garage vent. It worked for a while, but our dog got used to it and started ignoring it. The next step was to purchase a cheap windshield washer pump from Amazon (see below). The pump was connected to a gallon juice jug filled with water, and some plastic tubing was connected to the other end of the pump, routed through the garage vent and aimed at the gate. Finally, the pump was soldered to a 12V wall wart connected to an appliance module. Now in addition to the verbal correction, the pump gets turned on for 3 seconds, shooting a stream of water at the area behind the gate. He hates getting wet, so it's no surprise that the frequency and duration of barking has drastically declined :)

Follow up: Barking has declined from 3-5 times per day to maybe once a week!

It's Alive...Alive!!! (Our Infocast/Chumby that is)

Since the Chumby servers shutdown (and was switched to a stub service), the only available app is a clock. Some time ago, a Chumby user, Zurk, created an offline "firmware" for Chumbies to run without the Chumby servers in case they went down. He also released an app called octopus that downloaded all the apps off the Chumby servers, which was the thing to do when the servers were still up. With the downloaded apps and the "firmware" (more like some scripts and a local collection of Chumby files than actual firmware) Chumby nirvana could mostly be restored. I really only care about 1 app, Panel Builder 2, which runs on all our Infocasts. That rarely changes unless one of the kids wants to use some other app. I didn't even notice the Chumby servers went down because our Infocasts were still running the cached PB2. Slowly, one by one our Infocasts got reset to the default stub server clock. When my nightstand Infocast switched over, and I couldn't control the house from it, that was the last straw! I had to get this Zurk thing working. It was pretty much plug & play - just unzip the contents to a big enough USB disk. Panel Builder 2 was a private app for my testing only, so it couldn't have been grabbed by octopus. Since I wrote it, I just needed to stick the original .swf file on the USB drive, edit a few files to add the app and the Infocasts are useful again :) I've slowly been tweaking things and testing some other apps, but everything's back to normal. One caveat, it looks like apps that need configuration will not work. With PB2, I just hardcoded my server address and compiled a new .swf.

xpllib the CPU sucker

Recently, I had time to reinstall the OS on our media server, which was acting a little sluggish. I upgraded the RAM from 2GB to the max 4GB and installed Windows 7 x64 instead of the 32 bit that was on there. With the extra headroom, I migrated some xPL apps from my HA server to this one, but what I discovered was surprising. The media server has a relatively modern Pentium dual-core E5200, yet periodic bursts of xPL traffic would spike the CPU utilization up to near 100%. Three apps of mine (xPLGMail and 2 instances of xPLGVoice), would each suck up about 25% of the CPU. This didn't happen when those apps were on the E6420 Core2 HA server, but I did see something like this years ago when I was putting some xPL apps on HP T5700 thin clients. The culprit was the xpllib dll. The CPU intensive one is version 4.4.3663.31835, but some other apps I wrote using version 4.3.2737.14049 rarely use 1% of the E5200 CPU. I ended up taking a step back and recompiling xPLGMail and xPLGVoice with 4.3. Just like that, those apps never used more than 1% of the CPU. This reminded me of some years ago, when one of the more recent xPL devs started building a whole new xpllib (V5). At that time, I was working on the T5700s (still in use) so there's no way I would use an even fatter, more CPU intensive xpllib. (V5 at 320KB is almost 8 times bigger than 4.3 at 44KB). Not only that, the new V5 is not backwards compatible. I wasn't going to rewrite 20+ apps to "move forward." In fact, it looks like I will be migrating all my xPL apps backwards from xpllib 4.4 to 4.3. There's no need to waste CPU cycles for equivalent functionality. I'm slowly getting back to HA and it feels good.

Where Have I Been and Where is Chumby?

I've been here...working at another startup. It's been about 18 months and things really got hectic about December 2011 and hasn't let up since. All my HA work has been put on hold, with minor tweaks here and there. It's been my longest time away from HA since I started all this back around 1994, but I've done so much and everything just works. It's quite satisfying. Things are slowly easing up at work and I'm finding a little time and interest to do some more things, but it'll be a while before I'm able to commit as much time as before. One thing that really bums me out is the Chumby service has finally wound down. There's hope that it may come back, but for now there's just a clock widget. At some point, I will probably set up my own local server to serve widgets to our Infocasts, but there's no time for that right now. Chumby RIP for now.