AWS Internet of Things - IoT Test

AWS IoT provides secure, bi-directional communication between Internet-connected devices such as sensors, actuators, embedded micro-controllers, or smart appliances and the AWS Cloud. This enables you to collect telemetry data from multiple devices, and store and analyze the data. You can also create applications that enable your users to control these devices from their phones or tablets.

When using AWS IoT, devices are managed as things. A thing is a representation of a specific device or logical entity. It can be a physical device or sensor (for example, a light bulb or a switch on a wall). It can also be a logical entity like an instance of an application or physical entity that does not connect to AWS IoT but is related to other devices that do (for example, a car that has engine sensors or a control panel). Information about a thing is stored in the registry as JSON data.

These devices/things report their state by publishing messages, in JSON format, on MQTT topics. Each MQTT topic has a hierarchical name that identifies the device whose state is being updated. When a message is published on an MQTT topic, the message is sent to the AWS IoT MQTT message broker, which is responsible for sending all messages published on an MQTT topic to all clients subscribed to that topic. The message broker supports the use of the MQTT protocol to publish and subscribe and the HTTPS protocol to publish.

You can create rules that define one or more actions to perform based on the data in a message. For example, you can insert, update, or query a DynamoDB table or invoke a Lambda function. Rules use expressions to filter messages. When a rule matches a message, the rules engine invokes the action using the selected properties. Rules also contain an IAM role that grants AWS IoT permission to the AWS resources used to perform the action.

Each device has a shadow that stores and retrieves state information. Each item in the state information has two entries: the state last reported by the device and the desired state requested by an application. An application can request the current state information for a device. The shadow responds to the request by providing a JSON document with the state information (both reported and desired), metadata, and a version number. An application can control a device by requesting a change in its state. The shadow accepts the state change request, updates its state information, and sends a message to indicate the state information has been updated. The device receives the message, changes its state, and then reports its new state.

Figure 1 : How AWS IoT works

Typically, AWS IoT policies are used to enable devices to connect to the message broker and get authorized, so they can perform AWS IoT operations, such as subscribing or publishing to MQTT topics via the broker, or get, update, or delete a device's shadow. If any of these policy actions fail or experience errors, then devices may no longer be able to communicate with the AWS cloud. Also, where messages match configured rules, if the defined rule actions fail, the failure may disrupt communication between the devices and critical AWS services such as AWS Lambda, Amazon S3, Amazon DynamoDB, etc. Moreover, AWS also imposes limits on the AWS IOT service; if these limits are violated, it might adversely impact the overall performance of AWS IOT . To ensure the peak performance of AWS IOT therefore, administrators should continuously track the different types of requests to the message broker, capture errors, failures, and violations promptly, and rapidly initiate measures to fix the faults. This is where the AWS Internet of Things - IOT test helps!

By default, this test tracks the different types of requests (connection, publish, subscribe, unsubscribe, get/delete/update device shadow, etc.) to the message broker, and automatically discovers the protocols in use. For each protocol, the test then reports the count of requests that were successful, that violated the AWS IOT limits, and that which experienced errors. In the process, administrators can determine whether/not specific protocols are failing or are returning error responses frequently; such protocols could be suspect and may have to be investigated. Optionally, you can configure this test to report metrics for each rule or each action type configured for rules. This provides insights into errors (if any) in messages that match each rule and failed invocations of every action type.

Target of the test: Amazon Cloud

Agent deploying the test : A remote agent

Outputs of the test : One set of results for each protocol / rule / action type

First-level descriptor: AWS Region

Second-level descriptor: Protocol / rule / action type, depending upon the option chosen against the IOT Filter Name parameter.

Configurable parameters for the test
Parameter	Description
Test Period	How often should the test be executed.
Host	The host for which the test is to be configured.
Access Type	eG Enterprise monitors the AWS cloud using AWS API. By default, the eG agent accesses the AWS API using a valid AWS account ID, which is assigned a special role that is specifically created for monitoring purposes. Accordingly, the Access Type parameter is set to Role by default. Furthermore, to enable the eG agent to use this default access approach, you will have to configure the eG tests with a valid AWS Account ID to Monitor and the special AWS Role Name you created for monitoring purposes. Some AWS cloud environments however, may not support the role-based approach. Instead, they may allow cloud API requests only if such requests are signed by a valid Access Key and Secret Key. When monitoring such a cloud environment therefore, you should change the Access Type to Secret. Then, you should configure the eG tests with a valid AWS Access Key and AWS Secret Key. Note that the Secret option may not be ideal when monitoring high-security cloud environments. This is because, such environments may issue a security mandate, which would require administrators to change the Access Key and Secret Key, often. Because of the dynamicity of the key-based approach, Amazon recommends the Role-based approach for accessing the AWS API.
AWS Account ID to Monitor	This parameter appears only when the Access Type parameter is set to Role. Specify the AWS Account ID that the eG agent should use for connecting and making requests to the AWS API. To determine your AWS Account ID, follow the steps below: Login to the AWS management console. with your credentials. Click on your IAM user/role on the top right corner of the AWS Console. You will see a drop-down menu containing the Account ID (see Figure 2). Figure 2 : Identifying the AWS Account ID
AWS Role Name	This parameter appears when the Access Type parameter is set to Role. Specify the name of the role that you have specifically created on the AWS cloud for monitoring purposes. The eG agent uses this role and the configured Account ID to connect to the AWS Cloud and pull the required metrics. To know how to create such a role, refer to Creating a New Role.
AWS Access Key, AWS Secret Key, Confirm AWS Access Key, Confirm AWS Secret Key	These parameters appear only when the Access Type parameter is set to Secret.To monitor an Amazon cloud instance using the Secret approach, the eG agent has to be configured with the access key and secret key of a user with a valid AWS account. For this purpose, we recommend that you create a special user on the AWS cloud, obtain the access and secret keys of this user, and configure this test with these keys. The procedure for this has been detailed in the Obtaining an Access key and Secret key topic. Make sure you reconfirm the access and secret keys you provide here by retyping it in the corresponding Confirm text boxes.
Proxy Host and Proxy Port	In some environments, all communication with the AWS cloud and its regions could be routed through a proxy server. In such environments, you should make sure that the eG agent connects to the cloud via the proxy server and collects metrics. To enable metrics collection via a proxy, specify the IP address of the proxy server and the port at which the server listens against the Proxy Host and Proxy Port parameters. By default, these parameters are set to none , indicating that the eG agent is not configured to communicate via a proxy, by default.
Proxy User Name, Proxy Password, and Confirm Password	If the proxy server requires authentication, then, specify a valid proxy user name and password in the Proxy User Name and Proxy Password parameters, respectively. Then, confirm the password by retyping it in the Confirm Password text box. By default, these parameters are set to none, indicating that the proxy sever does not require authentication by default.
Proxy Domain and Proxy Workstation	If a Windows NTLM proxy is to be configured for use, then additionally, you will have to configure the Windows domain name and the Windows workstation name required for the same against the Proxy Domain and Proxy Workstation parameters. If the environment does not support a Windows NTLM proxy, set these parameters to none.
Exclude Region	Here, you can provide a comma-separated list of region names or patterns of region names that you do not want to monitor. For instance, to exclude regions with names that contain 'east' and 'west' from monitoring, your specification should be: east,west
IOT Filter Name	By default, this parameter is set to Protocol. In this case, this test will report metrics for every protocol. For each protocol, the test reports the count of requests that were successful, that violated the AWS IOT limits, and that which experienced errors. If required, you can override this default setting by setting the IOT Filter Name to one of the following: RuleName: For metrics on each rule that is configured, select this option. In this case, the test will only report the count of topics that match a rule condition and the count of errors (if any) in matching messages. ActionType: For metrics on every action type that is configured for a rule, select this option. In this case, the test will only report the count of successful and failed invocations of each action type.

Measurements made by the test
Measurement	Description	Measurement Unit	Interpretation
Connection request authorized errors	Indicates the number of connection requests made using that protocol, which could not be authorized by the message broker.	Number	This measure is reported only for each protocol - i.e., only if the IOT Filter Name parameter is set to 'Protocol'. The value 0 is desired for this measure. A high value is a cause for concern as it indicates that many connection requests made using a particular protocol were unauthorized.
Connection request client errors	Indicates the number of connection requests made using that protocol, which were rejected because the MQTT message did not meet the requirements defined in AWS IoT limits.	Number	This measure is reported only for each protocol - i.e., only if the IOT Filter Name parameter is set to 'Protocol'. Ideally, the value of this measure should be 0. A non-zero value indicates that a connection request has been rejected. Typically, AWS requires that a client ID in a connection request should not be more than 128 bytes in size and should consist of only UTF-8 encoded characters. Requests from clients that do not fulfill this requirement will be rejected.
Connection request server errors	Indicates the number of connection requests made using this protocol that failed because an internal error occurred.	Number	This measure is reported only for each protocol - i.e., only if the IOT Filter Name parameter is set to 'Protocol'. Ideally, the value of this measure should be 0. A non-zero value implies that for one/more connection requests made using that protocol, the message broker returned the error code 500.
Connection request success	Indicates the number of connection requests made using this protocol that resulted in successful connections to the message broker.	Number	This measure is reported only for each protocol - i.e., only if the IOT Filter Name parameter is set to 'Protocol'. A high value is desired for this measure.
Connection request throttled	Indicates the number of connection requests made using this protocol that were throttled because the client exceeded the allowed connect request rate.	Number	This measure is reported only for each protocol - i.e., only if the IOT Filter Name parameter is set to 'Protocol'. By default, AWS IoT limits an account to a maximum of 300 MQTT CONNECT requests per second. If this request rate is exceeded, then AWS throttles connection requests. You may want to observe the variations to the value of this measure over time and figure out if too many connection requests are throttled. If so, you may want to request for changing the connect request rate limit.
Delete thing shadow requests accepted	Indicates the number of DeleteThingShadow requests made using this protocol, which were processed successfully.	Number	This measure is reported only for each protocol - i.e., only if the IOT Filter Name parameter is set to 'Protocol'. A DeleteThingShadow request deletes the shadow of a specified thing.
Get thing shadow requests accepted	Indicates the number of GetThingShadow requests made using this protocol, which were processed successfully.	Number	This measure is reported only for each protocol - i.e., only if the IOT Filter Name parameter is set to 'Protocol'. A GetThingShadow request gets the shadow of a specified thing.
Successful ping messages	Indicates the number of ping messages sent using this protocol, which were successfully received by the message broker.	Number	This measure is reported only for each protocol - i.e., only if the IOT Filter Name parameter is set to 'Protocol'.
Publish in authorized errors	Indicates the number of inbound publish requests made using this protocol, which could not be authorized by the message broker.	Number	This measure is reported only for each protocol - i.e., only if the IOT Filter Name parameter is set to 'Protocol'. The value 0 is desired for this measure. A high value is a cause for concern as it indicates that many publish requests made using a particular protocol were unauthorized. Typically, publish requests that return the HTTP status code 401 are classified as unauthorized requests.
Publish in client errors	Indicates the number of inbound publish requests made using this protocol, which were rejected because the message did not meet the requirements defined in AWS IoT limits.	Number	This measure is reported only for each protocol - i.e., only if the IOT Filter Name parameter is set to 'Protocol'. Ideally, the value of this measure should be 0. A non-zero value indicates that a publish request has been rejected.
Publish in server errors	Indicates the number of inbound publish requests made using this protocol, which the message broker failed to process because an internal error occurred.	Number	This measure is reported only for each protocol - i.e., only if the IOT Filter Name parameter is set to 'Protocol'. Ideally, the value of this measure should be 0. A non-zero value implies that for one/more inbound publish requests made using that protocol, the message broker returned the error code 500.
Publish in success	Indicates the number of inbound publish requests made using this protocol that were successfully processed by the message broker.	Number	This measure is reported only for each protocol - i.e., only if the IOT Filter Name parameter is set to 'Protocol'. A high value is desired for this measure.
Publish in throttled	Indicates the number of inbound publish requests made using this protocol, which were throttled because the client exceeded the allowed inbound request rate.	Number	This measure is reported only for each protocol - i.e., only if the IOT Filter Name parameter is set to 'Protocol'. Typically, AWS IoT limits each client connection to 100 inbound publish requests per second. If this limit is exceeded, then subsequent inbound publish requests are throttled by AWS. You may want to observe variations to this measure over time and figure out if too many inbound publish requests are throttled. If so, you may want to request for changing the inbound publish request rate limit.
Publish out authorized errors	Indicates the number of outbound publish requests made using this protocol, which could not be authorized by the message broker.	Number	This measure is reported only for each protocol - i.e., only if the IOT Filter Name parameter is set to 'Protocol'. The value 0 is desired for this measure. A high value is a cause for concern as it indicates that many outbound publish requests made using a particular protocol were unauthorized. Typically, publish requests that return the HTTP status code 401 are classified as unauthorized requests.
Publish out client errors	Indicates the number of outbound publish requests made using this protocol, which were rejected because the message did not meet the requirements defined in AWS IoT limits.	Number	This measure is reported only for each protocol - i.e., only if the IOT Filter Name parameter is set to 'Protocol'. Ideally, the value of this measure should be 0. A non-zero value indicates that an outbound publish request has been rejected.
Publish out success	Indicates the number of outbound publish requests made using this protocol that were successfully processed by the message broker.	Number	This measure is reported only for each protocol - i.e., only if the IOT Filter Name parameter is set to 'Protocol'. A high value is desired for this measure.
Rules executed	Indicates the number of rules executed.	Number
Subscription request authorized errors	Indicates the number of subscription requests made using this protocol, which could not be authorized by the message broker.	Number	This measure is reported only for each protocol - i.e., only if the IOT Filter Name parameter is set to 'Protocol'. The value 0 is desired for this measure. A high value is a cause for concern as it indicates that many subscription requests made using a particular protocol were unauthorized. Typically, subscription requests that return the HTTP status code 401 are classified as unauthorized requests.
Subscription request client errors	Indicates the number of subscription requests made using this protocol, which were rejected because the message did not meet the requirements defined in AWS IoT limits.	Number	This measure is reported only for each protocol - i.e., only if the IOT Filter Name parameter is set to 'Protocol'. Ideally, the value of this measure should be 0. A non-zero value indicates that a subscription request has been rejected.
Subscription request server errors	Indicates the number of subscription requests made using this protocol, which the message broker failed to process because an internal error occurred.	Number	This measure is reported only for each protocol - i.e., only if the IOT Filter Name parameter is set to 'Protocol'. Ideally, the value of this measure should be 0. A non-zero value implies that for one/more inbound subscription requests made using that protocol, the message broker returned the error code 500.
Subscription request success	Indicates the number of subscription requests made using this protocol that were successfully processed by the message broker.	Number	This measure is reported only for each protocol - i.e., only if the IOT Filter Name parameter is set to 'Protocol'. A high value is desired for this measure.
Subscription request throttled	Indicates the number of subscription requests made using this protocol, which were throttled because the client exceeded the allowed subscription request rate.	Number	This measure is reported only for each protocol - i.e., only if the IOT Filter Name parameter is set to 'Protocol'. Typically, AWS IoT limits each client connection to subscribe to up to 50 subscriptions. A SUBSCRIBE request that pushes the total number of subscriptions past 50 results in the connection being disconnected. You may want to observe variations to this measure over time and figure out if too many subscription requests are throttled. If so, you may want to request for changing the subscription request rate limit.
Unsubscribe request client errors	Indicates the number of UNSUBSCRIBE requests made using this protocol, which were rejected because the UNSUBSCRIBE message did not meet the requirements defined in AWS IoT limits.	Number	This measure is reported only for each protocol - i.e., only if the IOT Filter Name parameter is set to 'Protocol'. Ideally, the value of this measure should be 0. A non-zero value indicates that an UNSUBSCRIBE request has been rejected.
Unsubscribe request server errors	Indicates the number of UNSUBSCRIBE requests made using this protocol, which the message broker failed to process because an internal error occurred.	Number	This measure is reported only for each protocol - i.e., only if the IOT Filter Name parameter is set to 'Protocol'. Ideally, the value of this measure should be 0. A non-zero value implies that for one/more UNSUBSCRIBE requests made using that protocol, the message broker returned the error code 500.
Unsubscribe request success	Indicates the number of UNSUBSCRIBE requests made using this protocol that were successfully processed by the message broker.	Number	This measure is reported only for each protocol - i.e., only if the IOT Filter Name parameter is set to 'Protocol'. A high value is desired for this measure.
Unsubscribe request throttled	Indicates the number of UNSUBSCRIBE requests made using this protocol, which were throttled because the client exceeded the allowed unsubscribe request rate.	Number	This measure is reported only for each protocol - i.e., only if the IOT Filter Name parameter is set to 'Protocol'. You may want to observe variations to this measure over time and figure out if too many UNSUBSCRIBE requests are throttled. If so, you may want to request for changing the request rate limit.
Update thing shadow requests accepted	Indicates the number of a UpdateThingShadow requests made using this protocol, which were received by the message broker.	Number	This measure is reported only for each protocol - i.e., only if the IOT Filter Name parameter is set to 'Protocol'. An UpdateThingShadow request updates the shadow of a specified thing.
Topic match	Indicates the number of incoming messages published on a topic on which this rule is listening.	Number	This measure is reported only for each rule - i.e., only if the IOT Filter Name parameter is set to 'RuleName'.
JSON parse errors	Indicates the number of JSON parse errors that occurred in messages published on a topic on which this rule is listening.	Number	This measure is reported only for each rule - i.e., only if the IOT Filter Name parameter is set to 'RuleName'. Ideally, the value of this measure should be 0.
Successful rule action invocations	Indicates the number of successful invocations of this rule action type.	Number	This measure is reported only for each rule action type - i.e., only if the IOT Filter Name parameter is set to 'ActionType'. Ideally, the value of this measure should be high.
Failed rule action invocations	Indicates the number of failed invocations of this rule action type.	Number	This measure is reported only for each rule action type - i.e., only if the IOT Filter Name parameter is set to 'ActionType'. Ideally, the value of this measure should be 0.