Skip to content

Commit 5fcb50e

Browse files
gthrone25gthrone25
committed
pushing nest stream example
1 parent d5178d8 commit 5fcb50e

File tree

3 files changed

+530
-0
lines changed

3 files changed

+530
-0
lines changed

‎streamdata-nest/README.md

+231
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,231 @@
1+
# Streaming your personal nest
2+
3+
There comes a point in testing data products when you become sick of using synthetic data and whatever you can find lying around on Kaggle. If you hit that point and start looking around your apartment for something REAL, you may just spot a Nest thermostat. This is capable of generating real data that you can stream into FeatureBase. This repo will walk you through this process, but there are some pre-requisites.
4+
5+
## Before you begin
6+
7+
* Purchase and setup a Nest Thermostat
8+
* [Follow these instructions](https://www.wouternieuwerth.nl/controlling-a-google-nest-thermostat-with-python/) to set up access to your nest data, which includes a purchase ($5) google developer access
9+
* Have a [FeatureBase Cloud account](https://cloud.featurebase.com/signup)
10+
* Have ready access to the following:
11+
* Nest Project ID
12+
* Nest access token
13+
* Nest refresh token
14+
* Nest client ID
15+
* FeatureBase username (email)
16+
* FeatureBase password
17+
* FeatureBase database id
18+
19+
All credit to the above blog for helping me get set up to access my nest's data!
20+
21+
## Setup python env
22+
23+
You need to have a python environment with the proper packages found in the requirements.txt
24+
25+
### Create using conda
26+
27+
```
28+
conda create --name nestbase --file requirements.txt
29+
```
30+
## Start
31+
32+
If you’re reading this section, you’ve completed the steps above and should feel accomplished already! You’ve bitten the bullet and paid Google for your own data, but you can now do whatever you want with it! First off, you need to get the device you want to pull data from. You can make a call to the smart device endpoint with your project to see all available devices:
33+
34+
```python
35+
def get_devices(self):
36+
"""Get all of the devices in your nest project
37+
38+
Returns:
39+
List: List of devices under your nest project
40+
"""
41+
url_get_devices = 'https://smartdevicemanagement.googleapis.com/v1/enterprises/' + self.project_id + '/devices'
42+
response = requests.get(url_get_devices, headers=self.headers)
43+
44+
return response.json()
45+
```
46+
47+
This walkthrough and code assumes you only have one thermostat device, but you can easily tweak the code to incorporate multiple devices if desired.
48+
49+
Once you have the correct device name, you can query it to get your device’s stats. This will return data that looks similar to the below payload:
50+
51+
```json
52+
{
53+
"type" : "sdm.devices.types.THERMOSTAT",
54+
"traits" : {
55+
"sdm.devices.traits.Connectivity" : {
56+
"status" : "ONLINE"
57+
},
58+
"sdm.devices.traits.Fan" : {
59+
"timerMode" : "ON",
60+
"timerTimeout" : "2019-05-10T03:22:54Z"
61+
},
62+
"sdm.devices.traits.Humidity" : {
63+
"ambientHumidityPercent" : 35.0
64+
},
65+
"sdm.devices.traits.Info" : {
66+
"customName" : "My device"
67+
},
68+
"sdm.devices.traits.Settings" : {
69+
"temperatureScale" : "CELSIUS"
70+
},
71+
"sdm.devices.traits.Temperature" : {
72+
"ambientTemperatureCelsius" : 23.0
73+
},
74+
"sdm.devices.traits.ThermostatEco" : {
75+
"availableModes" : ["MANUAL_ECO", "OFF"],
76+
"mode" : "MANUAL_ECO",
77+
"heatCelsius" : 20.0,
78+
"coolCelsius" : 22.0
79+
},
80+
"sdm.devices.traits.ThermostatHvac" : {
81+
"status" : "HEATING"
82+
},
83+
"sdm.devices.traits.ThermostatMode" : {
84+
"availableModes" : ["HEAT", "COOL", "HEATCOOL", "OFF"],
85+
"mode" : "COOL"
86+
},
87+
"sdm.devices.traits.ThermostatTemperatureSetpoint" : {
88+
"heatCelsius" : 20.0,
89+
"coolCelsius" : 22.0
90+
}
91+
}
92+
}
93+
```
94+
Detailed information about each trait can be found on [Google's docs](https://developers.google.com/nest/device-access/api/thermostat#json).
95+
96+
## Data modeling in FeatureBase
97+
98+
With a preview of the data, it’s time to model the data in FeatureBase. You can create a DDL statement with the traits of interest. Below is an example statement with familiar data types that elects the time of the thermostat reading as the table’s primary key (_id):
99+
100+
```sql
101+
CREATE TABLE nestbase (
102+
_id STRING,
103+
display_name STRING,
104+
device_type STRING,
105+
device_status STRING,
106+
fan_timer_mode STRING,
107+
thermostat_mode STRING,
108+
eco_mode STRING
109+
eco_heat_max_cel DECIMAL(6),
110+
eco_cool_min_cel DECIMAL(6),
111+
hvac_status STRING,
112+
temp_display_unit STRING,
113+
therm_heat_max_cel DECIMAL(6),
114+
therm_cool_min_cel DECIMAL(6),
115+
humidity_pct INT,
116+
ambient_temp_cel DECIMAL(6),
117+
measurement_ts TIMESTAMP
118+
);
119+
```
120+
121+
After inspecting a record, you realize how dependent your brain is on seeing Fahrenheit over Celsius. After going down an internal rabbit hole on why there is a metric system and an imperial system and questioning why the world can’t just get along, you decide that you want the ambient temperature to also show in Fahrenheit, so you add that column to the table:
122+
123+
```sql
124+
ALTER TABLE nestbase ADD ambient_temp_far DECIMAL(6);
125+
```
126+
127+
## Loading data
128+
129+
You are now ready to load data. For FeatureBase you can use [BULK INSERT statements](https://docs.featurebase.com/docs/sql-guide/statements/statement-insert-bulk/), which allow you to stream JSON data into your table. `BULK INSERT` gives you the flexibility to send 1 to n records, but for the examples that follow, each record will be sent individually. `BULK INSERT` allows for light data manipulation in the [TRANSFORM clause](https://docs.featurebase.com/docs/sql-guide/statements/statement-insert-bulk/#transform-clause-1), so you implement the temperature conversion there. An example of sending one record can be seen below:
130+
131+
```sql
132+
BULK INSERT INTO gt-nest-thermo (
133+
_id,
134+
display_name,
135+
device_type,
136+
device_status,
137+
fan_timer_mode,
138+
thermostat_mode,
139+
eco_mode,
140+
eco_heat_max_cel,
141+
eco_cool_min_cel,
142+
hvac_status,
143+
temp_display_unit,
144+
therm_heat_max_cel,
145+
therm_cool_min_cel,
146+
ambient_humidity_pct,
147+
ambient_temp_cel,
148+
measurement_ts,
149+
ambient_temp_far
150+
)
151+
MAP (
152+
'$["req_time"]' TIMESTAMP,
153+
'$["type"]' STRING,
154+
'$["traits"]["sdm.devices.traits.Humidity"]["ambientHumidityPercent"]' INT,
155+
'$["traits"]["sdm.devices.traits.Connectivity"]["status"]' STRING,
156+
'$["traits"]["sdm.devices.traits.Fan"]["timerMode"]' STRING,
157+
'$["traits"]["sdm.devices.traits.ThermostatMode"]["mode"]' STRING,
158+
'$["traits"]["sdm.devices.traits.ThermostatEco"]["mode"]' STRING,
159+
'$["traits"]["sdm.devices.traits.ThermostatEco"]["heatCelsius"]' DECIMAL(6),
160+
'$["traits"]["sdm.devices.traits.ThermostatEco"]["coolCelsius"]' DECIMAL(6),
161+
'$["traits"]["sdm.devices.traits.ThermostatHvac"]["status"]' STRING,
162+
'$["traits"]["sdm.devices.traits.Settings"]["temperatureScale"]' STRING,
163+
'$["traits"]["sdm.devices.traits.ThermostatTemperatureSetpoint"]["heatCelsius"]' DECIMAL(6),
164+
'$["traits"]["sdm.devices.traits.ThermostatTemperatureSetpoint"]["coolCelsius"]' DECIMAL(6),
165+
'$["traits"]["sdm.devices.traits.Temperature"]["ambientTemperatureCelsius"]' DECIMAL(6),
166+
'$["parentRelations"][0]["displayName"]' STRING
167+
)
168+
TRANSFORM(
169+
CAST(@0 as STRING),
170+
@14,
171+
@1,
172+
@3,
173+
@4,
174+
@5,
175+
@6,
176+
@7,
177+
@8,
178+
@9,
179+
@10,
180+
@11,
181+
@12,
182+
@2,
183+
@13,
184+
@0,
185+
(@13*9/5)+32)
186+
FROM '<record>'
187+
WITH
188+
BATCHSIZE 10000
189+
FORMAT 'NDJSON'
190+
INPUT 'STREAM'
191+
ALLOW_MISSING_VALUES;
192+
```
193+
194+
195+
## Putting it all together
196+
You are happy with the data model and are officially ready to start streaming data. You want to set and forget this and have data keep loading, so you need a couple of things: a connection to your Nest, a connection to FeatureBase, a way to refresh your connections, and a method to constantly pull and push data. Luckily [the nestbase.py script](/nestbase.py) does that all for you.
197+
198+
You are pretty smart, so you check the script out before running and find this will continuously run but only poll Nest every 6 seconds because Google limits device info to 10 QPM: https://developers.google.com/nest/device-access/project/limits . **Lastly, you correctly call into question the method in which secrets for Nest and FeatureBase are used and modify the script in accordance with your security needs.** You run the script (and use something like `caffeinate` when on a mac) and leave it be.
199+
200+
201+
## Running the script
202+
### Inputs
203+
* Have ready access to the following:
204+
* Nest Project ID
205+
* Nest access token
206+
* Nest refresh token
207+
* Nest client ID
208+
* FeatureBase username (email)
209+
* FeatureBase password
210+
* FeatureBase database id
211+
212+
### With a nest refresh token and a specific FeatureBase database
213+
214+
`python nestbase.py --project_id <> --access_token <> --refresh_token <> --client_id <> --client_secret <> --fb_user <> --fb_pw <> --fb_db <>`
215+
216+
### With a token that will expire within an hour
217+
218+
`python nestbase.py --project_id <> --access_token <> --fb_user <> --fb_pw <> --fb_db <>`
219+
220+
## Analyze the data
221+
222+
A couple days later you can return to play around with your recently loaded data. You have a couple of questions you want to answer:
223+
224+
1. What does the temperature and humidity look like at my place over time
225+
2. How variable is the temperature and humidity in my place?
226+
3. How long does it take for my place to cool down to my desired temperature?
227+
4. How often is my fan on during the day?
228+
5. When does my A/C turn on in relation to its target temperature
229+
230+
231+

‎streamdata-nest/ddl.sql

+19
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
CREATE TABLE nestbase (
2+
_id STRING,
3+
display_name STRING,
4+
device_type STRING,
5+
device_status STRING,
6+
fan_timer_mode STRING,
7+
thermostat_mode STRING,
8+
eco_mode STRING
9+
eco_heat_max_cel DECIMAL(6),
10+
eco_cool_min_cel DECIMAL(6),
11+
hvac_status STRING,
12+
temp_display_unit STRING,
13+
therm_heat_max_cel DECIMAL(6),
14+
therm_cool_min_cel DECIMAL(6),
15+
humidity_pct INT,
16+
ambient_temp_cel DECIMAL(6),
17+
measurement_ts TIMESTAMP
18+
);
19+

0 commit comments

Comments
 (0)