Do your AWS deployments match your Terraform definitions? Use SQL to find out.

In “How SQL can unify access to APIs” I made the case for SQL as a standard atmosphere through which to purpose about knowledge flowing from many various APIs. The important thing enabler of that state of affairs is Steampipe, a Postgres-based software with a rising suite of API plugins that map APIs to international tables in Postgres.

These APIs had been, initially, those supplied by AWS, Azure, and GCP. Such APIs are sometimes made extra accessible to builders by the use of wrappers like boto3. A standard SQL interface is arguably a better unifier of the sprawling API ecosystems inside these clouds, and that’s inarguably true in multicloud situations. With Postgres underneath the hood, by the way in which, you’re not restricted to SQL: You may hook Python or JavaScript or one other language to Postgres and leverage the widespread SQL interface from these languages too.

The Steampipe ecosystem then expanded with plugins for a lot of different providers together with GitHub, Google Workspace, IMAP, Jira, LDAP, Shodan, Slack, Stripe, and Zendesk. Becoming a member of throughout these APIs is a superpower finest confirmed by this instance that joins Amazon EC2 endpoints with Shodan vulnerabilities in simply 10 traces of very primary SQL.

choose
  a.instance_id,
  s.ports
  s.vulns
from
  aws_ec2_instance a
left be a part of
  shodan_host s on a.public_ip_address = s.ip
the place
  a.public_ip_address isn't null;

+---------------------+----------+--------------------+
| instance_id         | ports    | vulns              |
+---------------------+----------+--------------------+
| i-0dc60dd191cb84239 | null     | null               |
| i-042a51a815773780d | [80,22]  | null               |
| i-00cf426db9b8a58b6 | [22]     | null               |
| i-0e97f373db42dfa3f | [22,111] | ["CVE-2018-15919"] |
+---------------------+----------+--------------------+

Recordsdata are APIs too

However what’s an API, actually? Should it at all times entail HTTP requests to service endpoints? Extra broadly APIs are knowledge sources that are available different flavors too. Net pages are sometimes, nonetheless, de facto APIs. I’ve accomplished extra net scraping than I care to consider through the years and the talent stays helpful.

Recordsdata are additionally knowledge sources: configuration files (INI, YAML, JSON), infrastructure-as-code information (Terraform, CloudFormation), knowledge information (CSV). When plugins for these sources started to hitch the combination, Steampipe grew to become much more highly effective.

First got here the CSV plugin, which unlocked all kinds of helpful queries. Take into account, for instance, how we frequently fake spreadsheets are databases. In doing so we will assume there’s referential integrity when actually there isn’t. Should you export spreadsheet knowledge to CSV, you should use SQL to find those flawed assumptions. And that’s simply one of many infinite methods I can think about utilizing SQL to question the world’s main file format for knowledge change.

Then got here the Terraform plugin, which queries Terraform information to ask and reply questions like: “Which trails are usually not encrypted?”

choose
  identify,
  path
from
  terraform_resource
the place
  sort="aws_cloudtrail"
  and arguments -> 'kms_key_id' is null;

Utilizing the AWS plugin’s aws_cloudtrail_trail desk, we will ask and reply the identical query for deployed infrastructure, and return a end result set that you might UNION with the primary one.

choose
  identify,
  arn as path
from
  aws_cloudtrail_trail
the place
  kms_key_id is null;

Ideally the solutions will at all times be the identical. What you stated ought to be deployed, utilizing Terraform, ought to match what’s really deployed should you question AWS APIs. In the true world, in fact, upkeep and/or incident response can lead to configuration drift. Given a standard option to purpose over outlined and deployed infrastructure, we will handle such drift programmatically.

Belt and suspenders

For deployed infrastucture, Steampipe has lengthy supplied a set of mods that layer safety and compliance checks onto API-derived international tables. The AWS Compliance mod, for instance, offers benchmarks and controls to examine deployed infrastructure in opposition to eleven requirements and frameworks together with CIS, GDPR, HIPAA, NIST 800-53, and SOC 2.

IDG

With the appearance of the Terraform plugin it grew to become doable to create complementary mods, like Terraform AWS Compliance, that present the identical sorts of checks for outlined infrastructure.

IDG

Does what you outlined final month match what you deployed yesterday? A passable reply requires the flexibility to purpose over outlined and deployed infrastructure in a standard and frictionless means. SQL can’t take away all of the friction however it’s a strong solvent.