Home Assistant and MQTT

After getting Home Assistant up and running, the next thing I wanted to do was to add MQTT so I could connect sensors. I decided to use mosquitto for MQTT.

First to install mosquitto server, client and python mosquitto packages.

sudo apt-get install mosquitto mosquitto-clients python-mosquitto

Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
The following NEW packages will be installed:
The following packages will be upgraded:
libmosquitto1 mosquitto mosquitto-clients
3 upgraded, 1 newly installed, 0 to remove and 5 not upgraded.
Need to get 214 kB of archives.
After this operation, 148 kB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://mirrordirector.raspbian.org/raspbian/ jessie/main mosquitto-clients armhf 1.3.4-2+deb8u1 [39.5 kB]
Get:2 http://mirrordirector.raspbian.org/raspbian/ jessie/main libmosquitto1 armhf 1.3.4-2+deb8u1 [37.1 kB]
Get:3 http://mirrordirector.raspbian.org/raspbian/ jessie/main mosquitto armhf 1.3.4-2+deb8u1 [102 kB]
Get:4 http://mirrordirector.raspbian.org/raspbian/ jessie/main python-mosquitto all 1.3.4-2+deb8u1 [34.8 kB]
Fetched 214 kB in 2s (93.1 kB/s)
Reading changelogs... Done
(Reading database ... 35121 files and directories currently installed.)
Preparing to unpack .../mosquitto-clients_1.3.4-2+deb8u1_armhf.deb ...
Unpacking mosquitto-clients (1.3.4-2+deb8u1) over (1.3.4-2) ...
Preparing to unpack .../libmosquitto1_1.3.4-2+deb8u1_armhf.deb ...
Unpacking libmosquitto1 (1.3.4-2+deb8u1) over (1.3.4-2) ...
Preparing to unpack .../mosquitto_1.3.4-2+deb8u1_armhf.deb ...
Unpacking mosquitto (1.3.4-2+deb8u1) over (1.3.4-2) ...
Selecting previously unselected package python-mosquitto.
Preparing to unpack .../python-mosquitto_1.3.4-2+deb8u1_all.deb ...
Unpacking python-mosquitto (1.3.4-2+deb8u1) ...
Processing triggers for man-db ( ...
Processing triggers for systemd (215-17+deb8u7) ...
Setting up libmosquitto1 (1.3.4-2+deb8u1) ...
Setting up mosquitto-clients (1.3.4-2+deb8u1) ...
Setting up mosquitto (1.3.4-2+deb8u1) ...
Setting up python-mosquitto (1.3.4-2+deb8u1) ...
Processing triggers for libc-bin (2.19-18+deb8u9) ...

Now that it’s installed, lets set it up. First lets create the directory where it will keep it’s persistence db files, not forgetting to change the directory owner to the mosquitto user.

mkdir /var/lib/mosquitto/
sudo chown mosquitto:homeassistant /var/lib/mosquitto/ -R

Now lets update the configuration file. Below is what I’ve got in mine.

sudo nano /etc/mosquitto/mosquitto.conf

listener 1883
persistence true
persistence_location /var/lib/mosquitto/
persistence_file mosquitto.db
log_dest syslog
log_dest stdout
log_dest topic
log_type error
log_type warning
log_type notice
log_type information
connection_messages true
log_timestamp true
allow_anonymous false
password_file /etc/mosquitto/passwd

pid_file /var/run/mosquitto.pid

log_dest file /var/log/mosquitto/mosquitto.log

include_dir /etc/mosquitto/conf.d

Now to add some usernames/passwords.

This is how you’ll create the passwd file with the first user.

sudo mosquitto_passwd -c /etc/mosquitto/passwd username
prompt for passwd

After that, you add more users without the -c parameter, like this.

sudo mosquitto_passwd /etc/mosquitto/passwd ha
prompt for passwd

Now lets restart mosquitto.

sudo systemctl restart mosquitto

After the service has restarted, verify that mosquitto has started

pi@hassbian:~ $ sudo /etc/init.d/mosquitto status
● mosquitto.service - LSB: mosquitto MQTT v3.1 message broker
Loaded: loaded (/etc/init.d/mosquitto)
Active: active (running) since Sat 2017-06-03 23:29:23 AEST; 15s ago
Process: 30495 ExecStop=/etc/init.d/mosquitto stop (code=exited, status=0/SUCCESS)
Process: 30501 ExecStart=/etc/init.d/mosquitto start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/mosquitto.service
└─30507 /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf

Jun 03 23:29:23 hassbian systemd[1]: Starting LSB: mosquitto MQTT v3.1 message broker...
Jun 03 23:29:23 hassbian mosquitto[30501]: Starting network daemon:: mosquitto.
Jun 03 23:29:23 hassbian systemd[1]: Started LSB: mosquitto MQTT v3.1 message broker.
Jun 03 23:29:23 hassbian mosquitto[30507]: mosquitto version 1.3.4 (build date 2017-05-29 22:25:09+0000) starting
Jun 03 23:29:23 hassbian mosquitto[30507]: Config loaded from /etc/mosquitto/mosquitto.conf.
Jun 03 23:29:23 hassbian mosquitto[30507]: Opening ipv4 listen socket on port 1883.

Alright, now it’s up and running, lets give it a test.  I tested mine connecting to my raspberry pi with two SSH sessions, one to test subscribing to messages and one to test sending messages. You’ll need to update the IP Address, port number and username/password to suit you.

Subscribe to messages with topic test_mqtt

mosquitto_sub -t test_mqtt -u -P password -h -p 1883

Send a message to the topic test_mqtt

mosquitto_pub -d -t test_mqtt -m "Test Message" -h -p 1883 -u -P password

You should see the “Test Message” message arrive in your SSH session running the mosquitto subscribe.

Now lets add to Home Assistant

We now need to add some additional configuration to the Home Assistant configuration file for MQTT.

sudo nano /home/homeassistant/.homeassistant/configuration.yaml

You would add similar to the following, but customise it to IP Address and port number you are running mosquitto on (from the mosquitto configuration file) and a valid username/password.

broker: 172.16.X.Y
port: 1883
client_id: home-assistant-1
user: ha
passwd: hapassword

And then restart Home Assistant

This will get mosquitto up and running. You can now use MQTT with Home Assistant and send/receive message to MQTT sensors and clients.


Quickly retrieving data from Office 365 mailboxes 

Often we need to perform searches over many mailboxes to find just those few which match certain criteria.

One that I recently had was to find users which have a forward on their mailbox (set on the mailbox attribute) in Office 365.

One option I had was I could run:

get-mailbox -resultsize unlimited | where ($_.ForwardingSmtpAddress -like '*')

This will work.  However, if you’ve got a lot of users (e.g. over 100,000 users) it can be very slow.  But why?

As Office 365 is a shared platform, Microsoft throttle PowerShell commands.  They throttle PowerShell in a number of ways.  In this instance, by the amount of data that is sent back to your local PowerShell session.

But if I’ve only got 5 matching users out of thousands, surely that can’t be much data.. why is it throttled?

This is where it becomes important to know where the work for the PowerShell command is being done and when the data is being transferred.  As you are connecting to Office 365, some parts of the PowerShell command execute on remote server (Office 365 in our case), some is executed locally where you lauched PowerShell from and they pass data between them.

In the above example, the get-mailbox -resultsize unlimited retrieves that data from every mailbox.  This is done on Office 365 remote server and then it is all sent back to your local PowerShell session.  Once it arrives at your local PowerShell, it will execute the where ($_.ForwardingSmtpAddress -like '*') on the data.  The data is being transferred between the remote Office 365 and local server contains all the mailbox data, and of course due to the size of the data it gets throttled and slows down the command, taking longer to get the results.

We can speed this up!

The where executes locally, so we really want this to be executed on the remote server, decreasing the amount of data transferred back to your PowerShell session.  So how do we do that?  We can filter the results of the get-mailbox command using the -filter parameter and the command will only return what matches the filter and it performs it all on the remote server.

So we could change this example

get-mailbox -resultsize unlimited | where ($_.ForwardingSmtpAddress -like '*')


get-mailbox -resultsize unlimited -filter "ForwardingSmtpAddress -like '*'"

Prove it..

Unfortunately due to way these commands work, we can’t use -resultsize parameter to prove that these commands are faster. If we did use the -resultssize parameter e.g. -resultsize 100, the first example the command would only search the first 100 mailboxes while the updated version above would search mailboxes and return only first 100 forward results.

So in order to show the results, I’ll run this on a tenancy which contains over 300,000 mailboxes.  The original PowerShell command took over 2.5 hours while the new command took just over 10 minutes.


As you can see, the larger the number of mailboxes being searched the more benefit using -filter will give you.

If you have a small amount of users in your tenancy, then you probably won’t see very much difference in using the -filter parameter, however if you’re company expands or your scripts are used on larger Office 365 tenancies, using -filter may help your scripts get quicker results.