Question on SSM/Veeam interaction

[abelliniSIBA]

Hello guys,

I have one AWS account with production 2 VPCs.
AWS System Manager is deployed on every instance as patches automation tool.
I made the package distribution of VssComponents through AWS System Manager orchestration tool when i deployed the V2 and this was working fine.
After the upgrade to version 3, i'm receiving some warnings on 30% of my protected EC2 instances.

I was looking closer to this new message appearing which is the same on EC2s in warning status,
this happened on Snapshot creation part with Application-Aware Processing:

Failed to create VSS snapshot for instance "EC2ServerName":
Could not access the AWS API, therefore, VolumeId is not available.
Verify that your instance role has Describe-Instances permission

This behaviour makes me think is not a right issue.

I investigated further and i found in VAWS logs, something strange reference:
AWS SSM Agent try to lauch the _script id but with reference to an older id, not the last!!
I think this point could be an issue for VAWS working togheter with AWS System Manager.
It looks like AWS System Manager create additional commands ID whenever new commands are exchanged with the console,
but VAWS retry to launch the older reference saved on his db side!

I checked on documentation & Veeam KBs, i understood IAM role needed for Veeam server:
1. CF-Veeam AssumeRole;
2. CF-Veeam Backup&Restore;
3. (a custom one for Worker role as described);
+ Endpoint services configuration for backup/restore (S3, VPC, SSM)

To be honest i don't know exactly how my EC2 instances works togheter with Veeam&AWS System Manager

I checked on last user guide but i can't find the right place where the IAM role and SSM workflow point of view is explained.
I just want to make a step back and analyze what is needed or not.

If you guys have any suggestion or clear reference to easy understand the process,
it will be really appreciated.

Thanks
Alessandro

[nielsengelen]

Hi Alessandro, could you open a support case for this? From your description, it could indeed be related to old database information. For future reference, could you let us know the case ID?

[abelliniSIBA]

Case #04555941 — Vss interaction not working on some Ec2 snapshots

done

Thanks

Alessandro

[eeberg]

Hi Alessandro,

Here a few guidelines and steps for ensuring the VSS-enabled snapshots complete successfully.

1. The Windows instance must have the SSM agent installed and running. For system requirements and installation instructions for the SSM agent, visit the AWS Systems Manager documentation, https://docs.aws.amazon.com/systems-man ... l-win.html
2. The Windows instance must have an IAM instance profile attached that:
   * Allows Systems Manager to interact with the instance
   * Allows Systems Manager to create VSS-enabled snapshots
   For information on instance profiles and IAM policies required to create VSS-enabled snapshots, visit the AWS EC2 User Guide for Windows Instances, https://docs.aws.amazon.com/AWSEC2/late ... shots.html.
3. Ensure the IAM instance profile has the "AmazonSSMManagedInstanceCore" role attached.
4. Ensure the IAM instance profile has an IAM policy attached that allows VSS-enabled snapshots. See the following page for the JSON IAM policy for VSS-enabled snapshots, https://docs.aws.amazon.com/AWSEC2/late ... d-vss-role
5. The Windows instance must have the AWS VSS Components package (AwsVssComponents) installed. Installation instructions for the AWS VSS Components are in the AWS EC2 User Guide for Windows Instances, https://docs.aws.amazon.com/AWSEC2/late ... ss-package

Your specific issue might not be addressed by the guidelines above, however following them could help ensure that everything from the AWS side is configured correctly.

[abelliniSIBA]

Hi Eric,

thanks for the summary,
finally i found is not an issue related on Veeam Server.
It's a strange behaviour linked for some instances on AWS System manager.
It look like the association action with SSM is not working as expected for all instances.

From AWS console,
trying to run the command --> AWSEC2-CreateVssSnapshot gives me the same error i found on Veeam logs, but once again only for some of them.
My dumb action was missing this first debug step, maybe due to the long advise to stay home here #sorryguys.
I have to make a point zero and going through with my collegues.

I will keep you posted

Regards
Alessandro

[nielsengelen]

Hi Alessandro, glad to hear you managed to find the issue with help from Eric's tips. If there is anything else that comes up, just let us know.

[abelliniSIBA]

Hello guys,

quick update, i think i found the issue.
The problem is on prerequisites for Windows EC2.
In 2016 and 2019 version there is no need cause AMI is provided with all needed prerequisites for configuration steps.

Here are steps i followed to fix the issue on one Windows 2012r2 EC2 (affected OS version):

1. Uninstalled SSM Agent on EC2 (cleaned %ProgramData% caching);
2. Installed AWS Tools and SDK (msi setup containing Powershell extensions enabler);
3. Installed SSM agent;
4. Distributed AwsVssComponents package (here i think was now the point);
5. Run Command AWSEC2-CreateVssSnapshot and worked

I will replicate the procedure on all affected EC2 and i will let you know.

Cheers

Alessandro

[nielsengelen]

We'll look into this 2012 R2 specific issue and if needed, adjust our documentation with additional info on the SSM integration and caveats.

[abelliniSIBA]

Confirmed! It solved my issue.

Ec2 Windows 2012R2:
Installed AWSToolsAndSDKForNet_sdk-3.5.78.1_ps-4.1.5.0_tk-1.14.5.0.msi (AWS Tools for Windows Powershell component)
Re-run AWS-ConfigurePackage --> AwsVssComponents --> success
Running backup and VSS on snapshot succeded.

Better to have this spec in documentation.
Happy being able to give an active contribution on this topic.

Have a good Christmas everyone and stay safe.

Regards

Alessandro

[tcarracino]

Some of the things I do here are straight up Systems Manager and not Veeam.

If you are using a modern AMI (I mean later than 2008 R2) you have the SSM agent installed in all likely hood. You can check control panel and see.

If I want to tickle an instance (To update the agent) you can reboot it via the AWS Console which in my experience tends to initiate a version update.

that being said just because the agent is installed does not mean it is "Configured" (to run correctly on THAT instance)

So make sure your basics are setup on the account level in System Manager. Generally you create an IAM role and a policy and such.

Then you need to make sure that role is applied to the instance in question and make sure it shows up in systems manager under managed instances.

At that point if you get VSS error in Veeam you have to push the package AwsVssComponents using run command and you should be all set.

To confirm you can do a one time Run Command backup and use the VSS Snapshot document in Run Command to do it once.

Also if you test it with Run Command and it has MS SQL / Exchange/ Active Directory / or anything VSS you care about, ensure you configure an S3 bucket in the job.

That way you can go fetch the report out of the S3 bucket and see the entire job history for that one VSS snapshot and confirm everything got backed up correctly.

Hope that helps and LOVE VEEAM!

-Tom