Xiid Docs

Appearance

CloudWatch Integration ​

Integrate Xiid with CloudWatch to analyze your Xiid logs dynamically and get alerts of any issues in real-time.

Overview ​

Logs tend to be dispersed across separate machines and systems, making it difficult for administrators to be informed of software issues, user issues, and worst of all, malicious issues. Co-locating log files and being able to automatically analyze logs for concerning behavior is often a critical necessity in modern enterprise environments.

This guide will walk you through configuring your Xiid software logs so that they are automatically uploaded to CloudWatch Logs.

Prerequisites ​

This guide assumes that you have at least one Xiid software component installed (Collectors, Agents, STLinks, etc.).

This guide also assumes that you have a valid Amazon Web Services account and access to the CloudWatch Service.

You will also likely need AWS IAM permissions as well to either create a CloudWatch Service account or add an EC2 Role with CloudWatch permissions.

Permissions Setup ​

In order to push logs up to CloudWatch Logs, you need to have the relevant permissions available to access and write to Log Streams.

There are generally two paradigms for granting permissions for CloudWatch Logs. Which paradigm you choose is generally dependent on where your resources reside.

If your Xiid logs are on AWS EC2 instances (or other standard AWS services), it is highly recommended that you create an AWS Role for the machines/services, granting write access to CloudWatch Logs via Role-Based Permissions.

If you would like to upload Xiid logs from machines not in AWS, then it is recommended that you create a CloudWatch Service Account with relevant CloudWatch permissions.

Role-Based Permissions ​

Role-based CloudWatch permissions are the easiest and most straight-forward approach to granting the appropriate permissions to upload logs to CloudWatch. However, this strategy requires that all Xiid technology reside in AWS.

  • Sign in to your AWS Account through the AWS Console and navigate to the IAM service.
  • From the IAM service page, select the Roles option on the left-side navbar under "Access Management".
  • In the IAM Roles page, click the orange Create role button in the top right.
  • The first step in creating an AWS Role is to select Trusted Entity Type.
    • If you are setting up CloudWatch Log uploads for Xiid services running on an EC2 instance or other AWS service, select AWS Service under the Trusted Entity Type section. There are other entity types that may be relevant but for the purposes of this guide, we'll stick to EC2 instances.
  • Under the Use Case section, select the specific AWS Service that is running the Xiid software. (i.e. EC2)
  • Click the Next button to continue to Step 2.
    • Step 2 is where you actually define the permission policies that you would like to grant for those AWS resources that will assume the IAM Role.
  • Use the "Search" field to find AmazonEC2ReadOnlyAccess and check the box next to it. EC2 Readonly access is required to describe the instances, which is required by the CloudWatch Daemon.
  • Next, clear the "Search" field and search for CloudWatchLogsFullAccess. Normally, it is recommended to operate under least-permissions, but in the case of CloudWatch logs, there are not sufficiently granular permission sets for this use case.
  • Click the Next button to advance to step 3.
  • Provide a name and description for your new role (recommended name: Xiid CloudWatch Access). You can also optionally add anyTags` you would like to the Role.
  • Click the orange Create role button.

To add the role to existing EC2 instances:

  • Navigate to the EC2 Service in the AWS Console.
  • From the Instances table, check the box next to the EC2 instance that is running Xiid's software.
  • Next, click the Actions dropdown, click the Security option, and in the (third) context menu, select Modify IAM role.
  • In the Modify IAM role page, select the newly created Xiid CloudWatch Access role from the IAM role dropdown.
  • Click the Update IAM role button.

CloudWatch Service Account ​

The CloudWatch Service Account strategy works across any machine, whether it is in the AWS ecosystem or not. It requires a bit more work up-front, and may accrue a bit of technical debt in the event that AWS Access Keys need to be rotated.

These challenges are not uncommon for non-AWS-managed environments leveraging AWS services and the process is fairly seamless to execute.

  • Sign in to your AWS Account through the AWS Console and navigate to the IAM service.
  • From the IAM service page, select the Users option on the left-side navbar under Access Management.
  • On the IAM Users page, click the orange Create user button in the top right.
  • Provide a User name for the service account (recommended: xiid-log-svc), and do not check Provide user access to the AWS Management Console.
  • Click the Next button.
  • On Step 2, select the box labeled Attach policy directly.
  • Use the "Search" field to find AmazonEC2ReadOnlyAccess and check the box next to it. EC2 Readonly access is required to describe the instances, which is required by the CloudWatch Daemon.
  • Next, clear the "Search" field and search for CloudWatchLogsFullAccess. Normally, it is recommended to operate under least-permissions, but in the case of CloudWatch logs, there are not sufficiently granular permission sets for this use case.
  • After selecting the permissions, click the orange Next button in the bottom right.
  • Review the information for your service account, and then click Create user.

After the user is created, we need to generate Access Keys for the service account. The Access Keys will allow the CloudWatch Daemon access to the CloudWatch service (specifically for your AWS account).

  • Click the blue username text under the User name column in the Users page table.
  • On the User's account page, click the Security Credentials tab.
  • Under the Access keys section, click the Create access key button.
  • On the Use Case screen, select Other and then click the Next button.
  • Give a description for the Access Key and then click Create access key.
  • On the next screen, either download a .csv of your Access Key and Secret Key or copy and paste the keys somewhere safe for use in the CloudWatch Daemon setup section.

Install CloudWatch Agent ​

Next, you will need to install the AWS CloudWatch Agent on the machines that have the Xiid software. Even if the machines are in AWS (i.e. EC2), the CloudWatch Agent does not come pre-installed.

  • Sign in to the machine in which the Xiid Software is installed and open a web browser.
  • Download the latest version of the CloudWatch Agent onto the machine. Check the AWS CloudWatch Installation Guide for full instructions on installing the CloudWatch Agent on different platforms.
  • Open an Administrative Command Prompt and navigate to the directory where you downloaded the MSI file.
  • Enter the following command to install the AWS CloudWatch Agent:
batch
msiexec /i amazon-cloudwatch-agent.msi

After the command completes, you will have a (currently nonfunctioning) CloudWatch Agent installed on the machine.

Next, we'll configure the CloudWatch Agent to consume the logs on the machine.

Setup Amazon CLI Profiles ​

If you are deploying the CloudWatch Agent on a non-AWS machine, you will need to set up an instance profile using the service account Access/Secret Key set up in the previous section.

Follow these instructions to set up an instance profile: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/install-CloudWatch-Agent-on-premise.html

CloudWatch Configuration ​

The next step in setting up CloudWatch log uploads is to configure what logs to upload and where.

The CloudWatch Configuration Wizard is a prompt-based configuration setup that creates a json configuration file for you after answering some questions.

This guide will provide some defaults below, however these defaults should be considered after evaluating the prompt yourself. (i.e. Read the prompt, determine if the question is relevant to your use-case, if not, you can defer to the defaults provided).

Run the Configuration Wizard (recommended to run from an Administrator Command Line):

batch
cd "C:\Program Files\Amazon\AmazonCloudWatchAgent"
amazon-cloudwatch-agent-config-wizard.exe

Configuration Defaults ​

  • Which OS are you using?
    • Windows
  • EC2 or On-Premise hosting?
    • EC2 (or on-premise depending on use-case)
  • Turn on StatsD Daemon?
    • Yes (enables additional metrics for the machine, only available for EC2)
  • What port for StatsD Daemon?
    • 8125 (Just hit enter)
  • Collection interval for StatsD Daemon?
    • 10s (doesn't really matter, how often the metrics refresh)
  • StatsD Daemon Aggregation interval
    • 60s (again, not that important, how often it aggregates the above metrics)
  • Existing CloudWatch Config File?
    • No (unless you already did this step)
  • Monitor Host Metrics?
    • Yes (Up to you, this provides CPU/RAM, etc. metrics)
  • Monitor cpu metrics per core?
    • Yes (Again up to you)
  • Add EC2 Dimensions to metrics?
    • Yes (applicable particularly if you have multiple EC2 instances running CloudWatch Agent)
  • Aggregate EC2 dimensions?
    • Yes
  • High Resolution metrics?
    • 30s (How often do you want metrics refreshed, basically what delay do you want on metrics)
  • Which metric config do you want?
    • Basic
  • Satisfied with the configs?
    • Yes (hopefully)
  • Monitor customized log files?
    • YES (This one is Critically important to answer Yes)
  • Log File Path:
    • NOTE: Log Paths may change between versions
    • Please verify log path locations before entering in Wizard
    • IM Agent Log Path:
      • C:\ProgramData\Xiid\XIID-Agent\logs\events.json (Application Event Logs)
      • C:\ProgramData\Xiid\XIID-Agent\logs\log.json (Network Logs)
    • RDP Agent Log Path:
      • C:\ProgramData\Xiid\XIID-rdpagent\logs\events.json
    • STLink Log Path:
      • C:\ProgramData\Xiid\XIID-stlinkagent\logs\events.json
  • Log Group Name?
    • XiidServiceLogs (Suggested for all products)
  • Log Stream Name?
    • IM Agent:
      • IMEvents (For events.json log file)
      • IMNetworkLogs (for log.json file)
    • RDP Agent:
      • RDPEvents
    • STLink:
      • STEvents
  • Log Retention Policy
    • 30 days (option 7)
  • Any Additional log files?
    • Answer Yes if you have more log files on that machine to add. Repeat the steps starting at Monitor customized log files? for additional log files.
  • Monitor Windows event logs?
    • No (You can answer yes, I don't find the Windows logs terribly helpful)
  • Store the config in the SSM Parameter store?
    • No (Answer yes if you want to deploy the same config file on multiple EC2 instances)
  • Press enter to exit
    • We recommend pressing Enter

After that lengthy list of questions, we now have a config.json file stored in C:\Program Files\Amazon\AmazonCloudWatchAgent\config.json.

Launch the AWS CloudWatch Agent ​

With our configurations ready to go, we can start up the CloudWatch Agent!

Run the following command on Windows in an Administrative Powershell.

powershell
& "C:\Program Files\Amazon\AmazonCloudWatchAgent\amazon-cloudwatch-agent-ctl.ps1" -a fetch-config -m ec2 -s -c file:"C:\Program Files\Amazon\AmazonCloudWatchAgent\config.json"

After running the command, look for these three statements:

console
Configuration validation first phase succeeded
Configuration validation second phase succeeded
Configuration validation succeeded

Also be sure that the final statement says:

console
AmazonCloudWatchAgent has been started

And that's it!

From here, wait about 5 minutes and then navigate to CloudWatch in the AWS Console.

  • In the CloudWatch Console, select Log Groups on the left-side navbar under the Logs section.
  • After selecting the XiidServiceLogs group (or whatever you named the Log Group), you will see Log Streams named after the various types of logs we added from Xiid in the Configuration section.

Metrics and Alarms ​

TIP

We're expanding this guide to take things to the next step - generating metrics from Xiid log statements and enabling alarms on certain metrics, such as Failed Authentications. Stay tuned!

Copyright © 2024 Xiid Corporation