Let there be light

I moved into a place which used a remote control for turning the bedroom lights on and off. This lets me lie in bed and turn the lights off at night, however sometimes the remote control goes missing, or I forget to take it to the bed with me. It would be much more convenient if I could control it from my phone, this should be possible because my phone is much smarter than a simple remote control. The problem was figuring out how the remote control talked to the light. The remote didn’t have an infrared LED at the front like a TV remote control, hence I suspected it used radio. To find out which band it was using I whipped out an RTLSDR dongle I had lying around and tried the ISM bands. These are bands that are allowed for without licensing, for example the 2.4Ghz band (used for Wifi). I found that the remote was transmitting on 433.9 Mhz, using an On Of Keying (OOK) modulation. My phone can’t transmit on 433.9Mhz so this required a middleman. To get more clues I opened up the remote control and found it was 433.9Mhz transmitter connected to a simple microcontroller, there must be a receiver in the light that received the radio transmission from the remote and was connected to it’s own microcontroller that turned on the light.

I had an ESP8266 and I bought a transmitter receiver pair from eBay for $4. The transmitter and receiver were both extremely easy to connect to the ESP8266, all that was needed was VCC (power), ground and a data connection. Only slightly harder was getting a library which could make the ESP8266 talk to the transmitter and receiver, the RCSwitch library made talking to them a breeze. With the receiver connected I got it to decode the transmission from the remote control, this is so I could later replicate how the transmitter was talking to the light. I blasted the remote control in front of my receiver and found a 16 bit message, there were 4 bits dedicated to identifying the fan (allowing the remote to talk to 16 different fans) and a few other bits were used to tell the fan to turn off or turn on to low, medium or high or switch the light. There is no dedicated on and off button for the light, you can only switch the current state i.e. go from off to on or on to off.

With the remote codes sniffed I programmed the transmitter to repeat the codes. It worked on the first try and the light turned off and on. I now had a method of communicating with all the remote fan units, the next step was connecting my phone to the ESP. The ESP is so popular in hobbyist circles because it has inbuilt WiFi and bluetooth capability, it’s trivial to get connect it to a WiFI network , just add the password and AP name. The easiest way to communicate with the ESP is via a Web server, the default one is basic and I created a page that would serve up a buttons each corresponding to a remote control button. This system worked for a while but proved to be very slow to process web page inputs and would stop working after about 20 minutes. I couldn’t figure out why this was happening but I read that the async http server was much more stable and could handle more connections. I rewrote the ESP8266 web page code to use the async http server and it has been working really well since then, very responsive and hasn’t stopped working.

Where to BBQ?

In 2021 I wanted to be part of a GovHack team. I was inspired by Simon Victory‘s presentation on how experience as it sounded like a fun way to meet other people also interested in data and make something useful. Unfortunately the ACT went into lockdown a few weeks before the 2021 competition so it wasn’t possible to meet in person and I decided to tackle a problem by myself.

After looking over abundance of datasets available and match them the GovHack challenges, I decided to integrate ACT government data assets on public amenities such as BBQs, toilets, bubblers and playgrounds. The ACT government has datasets on the location and types of different amenities on their public data portal, they even have maps which chart the different locations of all the different amenities. Unfortunately the maps only show one type of amenity this means there are different maps for BBQs, toilets, dog parks and therefore you can’t see which BBQs have toilets near them or which parks have bubblers etc. .

Putting all the amenities on one map would be too messy, so I could only focus on one type of amenity. There are already apps for public toilets so I thought something for BBQs could be good. My idea was to create a map of BBQs and then add filters for the type of BBQ (gas, electric or wood) and if it had toilets, bubblers or a playground near by, the end result would be a map of BBQ locations over the ACT that met your needs.

It was a two stage process to build the tool, first was collecting and integrating all the ACT location data together . The second stage was actually presenting it in a visually pleasing manner. Collecting the data was simple, the ACT government provides a very easy to use API and clean data. There are a lot of BBQs which are clustered, instead of displaying these individually it made sense to group them together. I didn’t want to go through the bother of hand crafting groupings so I used a Density-based spatial clustering of applications with noise(DBSCAN) to find groups of BBQs, all I had to do was manually tweak two parameters, the distance between nodes and number of nodes in a cluster to get good groupings of BBQs. With groups of BBQs now worked out I had to figure out which had amenities close to them, this meant doing a cross join of all amenities with all BBQs and then figuring out if they were within the 100m distance threshold. As all the BBQs and the amenities were within the ACT, there wasn’t too much data and hence a cross join was feasible, however I don’t know how I would go about making this more efficient if I wanted to extend this to a bigger region, all of Australia.

With the data sufficiently wrangled I could move onto step two, map making. This is the first time I’ve made an interactive map and there are a plethora of options, I’m familiar with Python and I found the Bokeh visualisation framework was simple to use in Python and easy to export to the web. After a bit of tinkering around I was able to add custom filtering widgets, and thus was born my GovHack submission.

Bokeh Plot

You can find the original submission here, for a 46 hour submission I am pretty proud of myself, I even got an Honourable Mention as part of the Reimagining Government Services challenge. It needs more polish and it is mobile unfriendly, I could have made use of a designer.

Later on I made a version where all amenities are displayed, it’s a bit messy but provides information on even more amenities like dog parks and basketball courts, you can find it here.

The ACT government has done a really great job recording all the amenities and their locations, but their visualisation is analogous to a default excel chart, useful but so much utility is locked away. A little bit of work can provide a whole lot more value and I hope you find it useful. I sure did.

Time in the market vs Timing the market

Trying to time the perfect moment to buy and sell assets is difficult, most fund managers investing in a particular asset class don’t achieve the index of that class. The academic advice is to invest in a low fee fund that replicates the index, but it isn’t risk free, indexes can fall. Assume you went back in time, bought an asset for a fixed period of time, and then sold it once the time is expired, depending on what day you went back to you would have a different return.

This visualisation shows how changing the amount of time holding an asset changes the distribution of your returns.

How many weeks are you holding for?

- Data comes from Yahoo Finance(YF) which does not correctly report dividends.
 - Returns are based on close prices and assumes dividends are immediately reinvested. 
 - The average return and variation is the geometric mean and standard deviation of the returns.
 - Closing prices may sometimes be negative, it is capped at 1 cent to keep the results sensible
 - Return values are capped at the first percentile and ninety-ninth percentile to give reasonable results for highly volatile assets. 
 - Past performance doesn't guarantee future results.