1. Before you begin
Cloud-to-cloud integrations use device types to let the Google Assistant know what grammar should be used with a device. Device traits define the capabilities of a device type. A device inherits the states of each device trait added to an integration.
You can connect any supported traits to your chosen device type to customize the functionality of your users' devices. If you want to implement custom traits in your Actions that aren't currently available in the device schema, then the Modes and Toggles traits allow specific settings' control with a custom name that you define.
Beyond the basic control capability provided by types and traits, the Smart Home API has additional features to enhance the user experience. Error responses provide detailed user feedback when intents don't succeed. Secondary user verification extends those responses and adds additional security to the device trait of your choice. By sending specific error responses to challenge blocks issued from the Assistant, your Cloud-to-cloud integration can require additional authorization to complete a command.
Prerequisites
- Create a Cloud-to-cloud integration developer guide
- Smart Home Washer codelab
- Device types and traits developer guide
What you'll build
In this codelab, you'll deploy a prebuilt smart home integration with Firebase, then learn how to add nonstandard traits to the smart home washer for load size and turbo mode. You'll also implement error and exception reporting, and learn to enforce a verbal acknowledgement to turn on the washer using secondary user verification.
What you'll learn
- How to add the Modes and Toggles traits to your integration
- How to report errors and exceptions
- How to apply secondary user verification
What you'll need
- A web browser, such as Google Chrome
- An iOS or Android device with the Google Home app installed
- Node.js version 10.16 or later
- A Google Account
- A Google Cloud billing account
2. Getting started
Enable Activity controls
In order to use the Google Assistant, you must share certain activity data with Google. The Google Assistant needs this data to function properly; however, the requirement to share data is not specific to the SDK. To share this data, create a Google account if you don't already have one. You can use any Google account—it does not need to be your developer account.
Open the Activity Controls page for the Google account that you want to use with the Assistant.
Ensure the following toggle switches are enabled:
- Web & App Activity - In addition, be sure to select the Include Chrome history and activity from sites, apps, and devices that use Google services checkbox.
- Device Information
- Voice & Audio Activity
Create a Cloud-to-cloud Integration project
- Go to the Developer Console.
- Click Create Project, enter a name for the project, and click Create Project.
Select the Cloud-to-cloud Integration
On the Project Home in the Developer Console, select Add cloud-to-cloud integration under Cloud-to-cloud.
Install the Firebase CLI
The Firebase Command Line Interface (CLI) will allow you to serve your web apps locally and deploy your web app to Firebase hosting.
To install the CLI, run the following npm command from the terminal:
npm install -g firebase-tools
To verify that the CLI has been installed correctly, run:
firebase --version
Authorize the Firebase CLI with your Google account by running:
firebase login
Create a Firebase project
- Go to Firebase.
- Click Create a project and enter your project name.
- Check the agreement checkbox and click Continue. If there's no agreement checkbox, you may skip this step.
- Once your Firebase project is created, find the project ID. Go to Project Overview and click the settings icon > Project Settings.
- Your project is listed under the General tab.
Enable the HomeGraph API
The HomeGraph API enables the storage and querying of devices and their states within a user's Home Graph. To use this API, you must first open the Google Cloud console and enable the HomeGraph API.
In the Google Cloud console, make sure to select the project that matches your Actions <firebase-project-id>. Then, in the API Library screen for the HomeGraph API, click Enable.
3. Run the starter app
Now that you set up your development environment, you can deploy the starter project to verify everything is configured properly.
Get the source code
Click the following link to download the sample for this codelab on your development machine:
...or you can clone the GitHub repository from the command line:
git clone https://github.com/google-home/smarthome-traits.git
Unpack the downloaded zip file.
About the project
The starter project contains the following subdirectories:
public:A frontend UI to easily control and monitor the state of the smart washer.functions:A fully implemented cloud service that manages the smart washer with Cloud Functions for Firebase and Firebase Realtime Database.
The provided cloud fulfillment includes the following functions in index.js:
fakeauth: Authorization endpoint for account linkingfaketoken: Token endpoint for account linkingsmarthome: Smart home intent fulfillment endpointreportstate: Invokes the Home Graph API on device state changesrequestsync: Enables user device updates without requiring account relinking
Connect to Firebase
Navigate to the washer-start directory, then set up the Firebase CLI with your integration project:
cd washer-start firebase use <firebase-project-id>
Configure Firebase project
Initialize a Firebase project.
firebase init
Select the CLI features, Realtime Database, Functions, and the Hosting feature that includes Firebase Hosting.
? Which Firebase CLI features do you want to set up for this directory? Press Space to select features, then Enter to confirm your choices. ❯◉ Realtime Database: Configure a security rules file for Realtime Database and (optionally) provision default instance ◯ Firestore: Configure security rules and indexes files for Firestore ◉ Functions: Configure a Cloud Functions directory and its files ◉ Hosting: Configure files for Firebase Hosting and (optionally) set up GitHub Action deploys ◯ Hosting: Set up GitHub Action deploys ◯ Storage: Configure a security rules file for Cloud Storage ◯ Emulators: Set up local emulators for Firebase products ◯ Remote Config: Configure a template file for Remote Config ◯ Extensions: Set up an empty Extensions manifest
This will initialize the necessary APIs and features for your project.
When prompted, initialize Realtime Database. You can use the default location for the database instance.
? It seems like you haven't initialized Realtime Database in your project yet. Do you want to set it up? Yes ? Please choose the location for your default Realtime Database instance: us-central1
Since you are using the starter project code, choose the default file for the Security rules, and ensure you don't overwrite the existing database rules file.
? File database.rules.json already exists. Do you want to overwrite it with the Realtime Database Security Rules for <project-ID>-default-rtdb from the Firebase Console? No
If you are reinitializing your project, select Overwrite when asked if you want to initialize or overwrite a codebase.
? Would you like to initialize a new codebase, or overwrite an existing one? Overwrite
When configuring your Functions, you should use the default files, and ensure you don't overwrite the existing index.js and package.json files in the project sample.
? What language would you like to use to write Cloud Functions? JavaScript ? Do you want to use ESLint to catch probable bugs and enforce style? No ? File functions/package.json already exists. Overwrite? No ? File functions/index.js already exists. Overwrite? No
If you are reinitializing your project, select No when asked if you want to initialize or overwrite functions/.gitignore.
? File functions/.gitignore already exists. Overwrite? No
? Do you want to install dependencies with npm now? Yes
Finally, configure your Hosting setup to use the public directory in the project code, and use the existing index.html file. Select No when asked to use ESLint.
? What do you want to use as your public directory? public ? Configure as a single-page app (rewrite all urls to /index.html)? Yes ? Set up automatic builds and deploys with GitHub? No ? File public/index.html already exists. Overwrite? No
If ESLint was accidentally enabled, there are two methods available to disable it:
- Using the GUI, go to the
../functionsfolder under the project, select the hidden file.eslintrc.jsand delete it. Do not mistake it for the similarly named.eslintrc.json. - Using the command line:
cd functions rm .eslintrc.js
Deploy to Firebase
Now that you have installed the dependencies and configured your project, you are ready to run the app for the first time.
firebase deploy
This is the console output you should see:
... ✔ Deploy complete! Project Console: https://console.firebase.google.com/project/<firebase-project-id>/overview Hosting URL: https://<firebase-project-id>.web.app
This command deploys a web app, along with several Cloud Functions for Firebase.
Open the Hosting URL in your browser (https://<firebase-project-id>.web.app) to view the web app. You will see the following interface:
This web UI represents a third-party platform to view or modify device states. To begin populating your database with device information, click UPDATE. You won't see any changes on the page, but the current state of your washer will be stored in the database.
Now it's time to connect the cloud service you've deployed to the Google Assistant using the Developer Console.
Configure your Developer Console project
On the Develop tab, add a Display Name for your interaction. This name will appear in the Google Home app.
Under App branding, upload a png file for the app icon, sized 144 x 144px, and named
To enable Account linking use these account linking settings:
Client ID |
|
Client secret |
|
Authorization URL |
|
Token URL |
|
Under Cloud fulfillment URL, enter the URL for your cloud function that provides fulfillment for the smart home intents.
https://us-central1-
Click Save to save your project configuration, then click Next: Test to enable testing on your project.
Now you can begin implementing the webhooks necessary to connect the device state with the Assistant.
Link to Google Assistant
In order to test your Cloud-to-cloud integration, you need to link your project with a Google account. This enables testing through Google Assistant surfaces and the Google Home app that are signed in to the same account.
- On your phone, open the Google Assistant settings. Note that you should be logged in as the same account as in the console.
- Navigate to Google Assistant > Settings > Home Control (under Assistant).
- Click the search icon in the upper right.
- Search for your test app using the [test] prefix to find your specific test app.
- Select that item. The Google Assistant will then authenticate with your service and send a
SYNCrequest, asking your service to provide a list of devices for the user.
Open the Google Home app and verify that you can see your washer device.
Verify that you can control the washer by using voice commands in the Google Home app. You should also see the device state change in the frontend web UI of your cloud fulfillment.
Now that you have a basic washer deployed, you can customize the modes available on your device.
4. Add modes
The action.devices.traits.Modes trait enables a device to have an arbitrary number of settings for a mode, of which only one can be set at a time. You'll add a mode to the washer to define the size of the laundry load: small, medium, or large.
Update SYNC response
You need to add information about the new trait to your SYNC response in functions/index.js. This data appears in the traits array and attributes object as shown in the following code snippet.
index.js
app.onSync(body => {
return {
requestId: 'ff36a3cc-ec34-11e6-b1a0-64510650abcf',
payload: {
agentUserId: USER_ID,
devices: [{
id: 'washer',
type: 'action.devices.types.WASHER',
traits: [
'action.devices.traits.OnOff',
'action.devices.traits.StartStop',
'action.devices.traits.RunCycle',
// Add Modes trait
'action.devices.traits.Modes',
],
name: { ... },
deviceInfo: { ... },
attributes: {
pausable: true,
//Add availableModes
availableModes: [{
name: 'load',
name_values: [{
name_synonym: ['load'],
lang: 'en',
}],
settings: [{
setting_name: 'small',
setting_values: [{
setting_synonym: ['small'],
lang: 'en',
}]
}, {
setting_name: 'medium',
setting_values: [{
setting_synonym: ['medium'],
lang: 'en',
}]
}, {
setting_name: 'large',
setting_values: [{
setting_synonym: ['large'],
lang: 'en',
}]
}],
ordered: true,
}],
},
}],
},
};
});
Add new EXECUTE intent commands
In your EXECUTE intent, add the action.devices.commands.SetModes command as shown in the following code snippet.
index.js
const updateDevice = async (execution,deviceId) => {
const {params,command} = execution;
let state, ref;
switch (command) {
case 'action.devices.commands.OnOff':
state = {on: params.on};
ref = firebaseRef.child(deviceId).child('OnOff');
break;
case 'action.devices.commands.StartStop':
state = {isRunning: params.start};
ref = firebaseRef.child(deviceId).child('StartStop');
break;
case 'action.devices.commands.PauseUnpause':
state = {isPaused: params.pause};
ref = firebaseRef.child(deviceId).child('StartStop');
Break;
// Add SetModes command
case 'action.devices.commands.SetModes':
state = {load: params.updateModeSettings.load};
ref = firebaseRef.child(deviceId).child('Modes');
break;
}
Update QUERY response
Next, update your QUERY response to report the washer's current state.
Add the updated changes to the queryFirebase and queryDevice functions to obtain the state as stored in the Realtime Database.
index.js
const queryFirebase = async (deviceId) => {
const snapshot = await firebaseRef.child(deviceId).once('value');
const snapshotVal = snapshot.val();
return {
on: snapshotVal.OnOff.on,
isPaused: snapshotVal.StartStop.isPaused,
isRunning: snapshotVal.StartStop.isRunning,
// Add Modes snapshot
load: snapshotVal.Modes.load,
};
}
const queryDevice = async (deviceId) => {
const data = await queryFirebase(deviceId);
return {
on: data.on,
isPaused: data.isPaused,
isRunning: data.isRunning,
currentRunCycle: [{ ... }],
currentTotalRemainingTime: 1212,
currentCycleRemainingTime: 301,
// Add currentModeSettings
currentModeSettings: {
load: data.load,
},
};
};
Update Report State
Finally, update your reportstate function to report the washer's current load setting to Home Graph.
index.js
const requestBody = {
requestId: 'ff36a3cc', /* Any unique ID */
agentUserId: USER_ID,
payload: {
devices: {
states: {
/* Report the current state of your washer */
[context.params.deviceId]: {
on: snapshot.OnOff.on,
isPaused: snapshot.StartStop.isPaused,
isRunning: snapshot.StartStop.isRunning,
// Add currentModeSettings
currentModeSettings: {
load: snapshot.Modes.load,
},
},
},
},
},
};
Deploy to Firebase
Run the following command to deploy the updated integration:
firebase deploy --only functions
After deployment completes, navigate to the web UI and click the Refresh 
button in the toolbar. That triggers a Request Sync so that the Assistant receives the updated SYNC response data.
Now you can give a command to set the mode of the washer, such as:
"Hey Google, set the washer load to large."
Additionally, you can ask questions about your washer, such as:
"Hey Google, what is the washer load?"
5. Add toggles
The action.devices.traits.Toggles trait represents named aspects of a device that have a true or false state, such as whether the washer is in turbo mode.
Update SYNC response
In your SYNC response, you need to add information about the new device trait. It will appear in the traits array and attributes object as shown in the following code snippet.
index.js
app.onSync(body => {
return {
requestId: 'ff36a3cc-ec34-11e6-b1a0-64510650abcf',
payload: {
agentUserId: USER_ID,
devices: [{
id: 'washer',
type: 'action.devices.types.WASHER',
traits: [
'action.devices.traits.OnOff',
'action.devices.traits.StartStop',
'action.devices.traits.RunCycle',
'action.devices.traits.Modes',
// Add Toggles trait
'action.devices.traits.Toggles',
],
name: { ... },
deviceInfo: { ... },
attributes: {
pausable: true,
availableModes: [{
name: 'load',
name_values: [{
name_synonym: ['load'],
lang: 'en'
}],
settings: [{ ... }],
ordered: true,
}],
//Add availableToggles
availableToggles: [{
name: 'Turbo',
name_values: [{
name_synonym: ['turbo'],
lang: 'en',
}],
}],
},
}],
},
};
});
Add new EXECUTE intent commands
In your EXECUTE intent, add the action.devices.commands.SetToggles command as shown in the following code snippet.
index.js
const updateDevice = async (execution,deviceId) => {
const {params,command} = execution;
let state, ref;
switch (command) {
case 'action.devices.commands.OnOff':
state = {on: params.on};
ref = firebaseRef.child(deviceId).child('OnOff');
break;
case 'action.devices.commands.StartStop':
state = {isRunning: params.start};
ref = firebaseRef.child(deviceId).child('StartStop');
break;
case 'action.devices.commands.PauseUnpause':
state = {isPaused: params.pause};
ref = firebaseRef.child(deviceId).child('StartStop');
break;
case 'action.devices.commands.SetModes':
state = {load: params.updateModeSettings.load};
ref = firebaseRef.child(deviceId).child('Modes');
break;
// Add SetToggles command
case 'action.devices.commands.SetToggles':
state = {Turbo: params.updateToggleSettings.Turbo};
ref = firebaseRef.child(deviceId).child('Toggles');
break;
}
Update QUERY response
Finally, you need to update your QUERY response to report the washer's turbo mode. Add the updated changes to the queryFirebase and queryDevice functions to obtain the toggle state as stored in the Realtime Database.
index.js
const queryFirebase = async (deviceId) => {
const snapshot = await firebaseRef.child(deviceId).once('value');
const snapshotVal = snapshot.val();
return {
on: snapshotVal.OnOff.on,
isPaused: snapshotVal.StartStop.isPaused,
isRunning: snapshotVal.StartStop.isRunning,
load: snapshotVal.Modes.load,
// Add Toggles snapshot
Turbo: snapshotVal.Toggles.Turbo,
};
}
const queryDevice = async (deviceId) => {
const data = queryFirebase(deviceId);
return {
on: data.on,
isPaused: data.isPaused,
isRunning: data.isRunning,
currentRunCycle: [{ ... }],
currentTotalRemainingTime: 1212,
currentCycleRemainingTime: 301,
currentModeSettings: {
load: data.load,
},
// Add currentToggleSettings
currentToggleSettings: {
Turbo: data.Turbo,
},
};
};
Update Report State
Finally, update your reportstate function to report to Home Graph whether the washer is set to turbo.
index.js
const requestBody = {
requestId: 'ff36a3cc', /* Any unique ID */
agentUserId: USER_ID,
payload: {
devices: {
states: {
/* Report the current state of your washer */
[context.params.deviceId]: {
on: snapshot.OnOff.on,
isPaused: snapshot.StartStop.isPaused,
isRunning: snapshot.StartStop.isRunning,
currentModeSettings: {
load: snapshot.Modes.load,
},
// Add currentToggleSettings
currentToggleSettings: {
Turbo: snapshot.Toggles.Turbo,
},
},
},
},
},
};
Deploy to Firebase
Run the following command to deploy the updated functions:
firebase deploy --only functions
Click the Refresh button in the web UI to trigger a Request Sync after deployment completes.
You can now give a command to set the washer to turbo mode by saying:
"Hey Google, turn on turbo for the washer."
You can also check if your washer is already in turbo mode by asking:
"Hey Google, is my washer in turbo mode?"
6. Reporting errors and exceptions
Error handling in your Cloud-to-cloud integration enables you to report to users when issues cause EXECUTE and QUERY responses to fail. The notifications create a more positive user experience for your users when they interact with your smart device and integration.
Any time that an EXECUTE or QUERY request fails, your integration should return an error code. If, for example, you wanted to throw an error when a user attempts to start the washer with the lid open, then your EXECUTE response would look like the following code snippet:
{
"requestId": "ff36a3cc-ec34-11e6-b1a0-64510650abcf",
"payload": {
"commands": [
{
"ids": [
"456"
],
"status": "ERROR",
"errorCode": "deviceLidOpen"
}
]
}
}
Now, when a user asks to start the washer, the Assistant responds by saying:
"The lid is open on the washer. Please close it and try again."
Exceptions are similar to errors, but indicate when an alert is associated with a command, which may or may not block successful execution. An exception can provide related information using the StatusReport trait, such as battery level or recent state change. Non-blocking exception codes are returned along with a SUCCESS status, while blocking exception codes are returned with an EXCEPTIONS status.
An example response with an exception is in the following code snippet:
{
"requestId": "ff36a3cc-ec34-11e6-b1a0-64510650abcf",
"payload": {
"commands": [{
"ids": ["123"],
"status": "SUCCESS",
"states": {
"online": true,
"isPaused": false,
"isRunning": false,
"exceptionCode": "runCycleFinished"
}
}]
}
}
The Assistant responds by saying:
"The washer has finished running."
To add error reporting for your washer, open functions/index.js and add the error class definition as seen in the following code snippet:
index.js
app.onQuery(async (body) => {...});
// Add SmartHome error handling
class SmartHomeError extends Error {
constructor(errorCode, message) {
super(message);
this.name = this.constructor.name;
this.errorCode = errorCode;
}
}
Update the execute response to return the error code and the error status:
index.js
const executePromises = [];
const intent = body.inputs[0];
for (const command of intent.payload.commands) {
for (const device of command.devices) {
for (const execution of command.execution) {
executePromises.push(
updateDevice(execution, device.id)
.then((data) => {
...
})
//Add error response handling
.catch((error) => {
functions.logger.error('EXECUTE', device.id, error);
result.ids.push(device.id);
if (error instanceof SmartHomeError) {
result.status = 'ERROR';
result.errorCode = error.errorCode;
}
})
);
}
}
}
The Assistant can now tell your users about any error code that you report. You'll see a specific example in the next section.
7. Add secondary user verification
You should implement secondary user verification in your integration if your device has any modes that need to be secured or should be limited to a particular group of authorized users, such as a software update or lock disengage.
You can implement secondary user verification on all device types and traits, customizing whether the security challenge occurs every time or specific criteria needs to be met.
There are three supported challenge types:
Nochallenge—A request and response that doesn't use an authentication challenge (this is the default behavior)ackNeeded—A secondary user verification that requires explicit acknowledgement (yes or no)pinNeeded—A secondary user verification that requires a personal identification number (PIN)
For this codelab, add an ackNeeded challenge to the command for turning on the washer and the functionality to return an error if the secondary verification challenge fails.
Open functions/index.js, and add an error class definition that returns the error code and challenge type as seen in the following code snippet:
index.js
class SmartHomeError extends Error { ... }
// Add secondary user verification error handling
class ChallengeNeededError extends SmartHomeError {
/**
* Create a new ChallengeNeededError
* @param {string} suvType secondary user verification challenge type
*/
constructor(suvType) {
super('challengeNeeded', suvType);
this.suvType = suvType;
}
}
You also need to update the execution response to return the challengeNeeded error as follows:
index.js
const executePromises = [];
const intent = body.inputs[0];
for (const command of intent.payload.commands) {
for (const device of command.devices) {
for (const execution of command.execution) {
executePromises.push(
updateDevice(execution, device.id)
.then((data) => {
...
})
.catch((error) => {
functions.logger.error('EXECUTE', device.id, error);
result.ids.push(device.id);
if (error instanceof SmartHomeError) {
result.status = 'ERROR';
result.errorCode = error.errorCode;
//Add error response handling
if (error instanceof ChallengeNeededError) {
result.challengeNeeded = {
type: error.suvType
};
}
}
})
);
}
}
}
Finally, modify updateDevice to require the explicit acknowledgment to turn the washer on or off.
index.js
const updateDevice = async (execution,deviceId) => {
const {challenge,params,command} = execution; //Add secondary user challenge
let state, ref;
switch (command) {
case 'action.devices.commands.OnOff':
//Add secondary user verification challenge
if (!challenge || !challenge.ack) {
throw new ChallengeNeededError('ackNeeded');
}
state = {on: params.on};
ref = firebaseRef.child(deviceId).child('OnOff');
break;
...
}
return ref.update(state)
.then(() => state);
};
Deploy to Firebase
Run the following command to deploy the updated function:
firebase deploy --only functions
After deploying the updated code, you must verbally acknowledge the action when you ask the Assistant to turn your washer on or off, like this:
You: "Hey Google, turn on the washer."
The Assistant: "Are you sure you want to turn on the washer?"
You: "Yes."
You can also see a detailed response for each step of the secondary user verification flow by opening your Firebase logs.
8. Congratulations
Congratulations! You extended the features of Cloud-to-cloud integrations through the Modes and Toggles traits, and secured their execution through secondary user verification.
Learn more
Here are some ideas you can implement to go deeper:
- Add local execution capabilities to your devices.
- Use a different secondary user verification challenge type to modify your device state.
- Update the
RunCycletrait QUERY response to update dynamically. - Explore this GitHub sample.