Skip to content

T
terraform-aws-vpc
Commit 2423e4ff authored by drewmullen's avatar drewmullen Committed by GitHub
Browse files
Options

feat: Added support for VPC Flow Logs in Parquet format (#700)

parent 96d22b8c
Hide whitespace changes
Inline Side-by-side
Showing with 150 additions and 50 deletions
.github/contributing.md 0 → 100644
View file @ 2423e4ff
# Contributing
When contributing to this repository, please first discuss the change you wish to make via issue,
email, or any other method with the owners of this repository before making a change.
Please note we have a code of conduct, please follow it in all your interactions with the project.
## Pull Request Process
1. Update the README.md with details of changes including example hcl blocks and [example files](./examples) if appropriate.
2. Run pre-commit hooks `pre-commit run -a`.
3. Once all outstanding comments and checklist items have been addressed, your contribution will be merged! Merged PRs will be included in the next release. The terraform-aws-vpc maintainers take care of updating the CHANGELOG as they merge.
## Checklists for contributions
- [ ] Add [semantics prefix](#semantic-pull-requests) to your PR or Commits (at least one of your commit groups)
- [ ] CI tests are passing
- [ ] README.md has been updated after any changes to variables and outputs. See https://github.com/terraform-aws-modules/terraform-aws-vpc/#doc-generation
- [ ] Run pre-commit hooks `pre-commit run -a`
## Semantic Pull Requests
To generate changelog, Pull Requests or Commits must have semantic and must follow conventional specs below:
- `feat:` for new features
- `fix:` for bug fixes
- `improvement:` for enhancements
- `docs:` for documentation and examples
- `refactor:` for code refactoring
- `test:` for tests
- `ci:` for CI purpose
- `chore:` for chores stuff
The `chore` prefix skipped during changelog generation. It can be used for `chore: update changelog` commit message by example.
README.md
View file @ 2423e4ff
......@@ -120,6 +120,14 @@ You can add additional tags with `intra_subnet_tags` as with other subnet types.
VPC Flow Log allows to capture IP traffic for a specific network interface (ENI), subnet, or entire VPC. This module supports enabling or disabling VPC Flow Logs for entire VPC. If you need to have VPC Flow Logs for subnet or ENI, you have to manage it outside of this module with [aws_flow_log resource](https://www.terraform.io/docs/providers/aws/r/flow_log.html).
### VPC Flow Log Examples
By default `file_format` is `plain-text`. You can also specify `parquet` to have logs written in Apache Parquet format.
```
flow_log_file_format = "parquet"
```
### Permissions Boundary
If your organization requires a permissions boundary to be attached to the VPC Flow Log role, make sure that you specify an ARN of the permissions boundary policy as `vpc_flow_log_permissions_boundary` argument. Read more about required [IAM policy for publishing flow logs](https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs-cwl.html#flow-logs-iam).
......@@ -183,19 +191,25 @@ It is possible to integrate this VPC module with [terraform-aws-transit-gateway
- [Manage Default VPC](https://github.com/terraform-aws-modules/terraform-aws-vpc/tree/master/examples/manage-default-vpc)
- [Few tests and edge case examples](https://github.com/terraform-aws-modules/terraform-aws-vpc/tree/master/examples/issues)
## Contributing
Report issues/questions/feature requests on in the [issues](https://github.com/terraform-aws-modules/terraform-aws-vpc/issues/new) section.
Full contributing [guidelines are covered here](.github/contributing.md).
<!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
## Requirements
| Name | Version |
|------|---------|
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.12.31 |
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.38 |
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.13.1 |
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.63 |
## Providers
| Name | Version |
|------|---------|
| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 3.38 |
| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 3.63 |
## Modules
......@@ -370,8 +384,11 @@ No modules.
| <a name="input_flow_log_cloudwatch_log_group_retention_in_days"></a> [flow\_log\_cloudwatch\_log\_group\_retention\_in\_days](#input\_flow\_log\_cloudwatch\_log\_group\_retention\_in\_days) | Specifies the number of days you want to retain log events in the specified log group for VPC flow logs. | `number` | `null` | no |
| <a name="input_flow_log_destination_arn"></a> [flow\_log\_destination\_arn](#input\_flow\_log\_destination\_arn) | The ARN of the CloudWatch log group or S3 bucket where VPC Flow Logs will be pushed. If this ARN is a S3 bucket the appropriate permissions need to be set on that bucket's policy. When create\_flow\_log\_cloudwatch\_log\_group is set to false this argument must be provided. | `string` | `""` | no |
| <a name="input_flow_log_destination_type"></a> [flow\_log\_destination\_type](#input\_flow\_log\_destination\_type) | Type of flow log destination. Can be s3 or cloud-watch-logs. | `string` | `"cloud-watch-logs"` | no |
| <a name="input_flow_log_file_format"></a> [flow\_log\_file\_format](#input\_flow\_log\_file\_format) | (Optional) The format for the flow log. Valid values: `plain-text`, `parquet`. | `string` | `"plain-text"` | no |
| <a name="input_flow_log_hive_compatible_partitions"></a> [flow\_log\_hive\_compatible\_partitions](#input\_flow\_log\_hive\_compatible\_partitions) | (Optional) Indicates whether to use Hive-compatible prefixes for flow logs stored in Amazon S3. | `bool` | `false` | no |
| <a name="input_flow_log_log_format"></a> [flow\_log\_log\_format](#input\_flow\_log\_log\_format) | The fields to include in the flow log record, in the order in which they should appear. | `string` | `null` | no |
| <a name="input_flow_log_max_aggregation_interval"></a> [flow\_log\_max\_aggregation\_interval](#input\_flow\_log\_max\_aggregation\_interval) | The maximum interval of time during which a flow of packets is captured and aggregated into a flow log record. Valid Values: `60` seconds or `600` seconds. | `number` | `600` | no |
| <a name="input_flow_log_per_hour_partition"></a> [flow\_log\_per\_hour\_partition](#input\_flow\_log\_per\_hour\_partition) | (Optional) Indicates whether to partition the flow log per hour. This reduces the cost and response time for queries. | `bool` | `false` | no |
| <a name="input_flow_log_traffic_type"></a> [flow\_log\_traffic\_type](#input\_flow\_log\_traffic\_type) | The type of traffic to capture. Valid values: ACCEPT, REJECT, ALL. | `string` | `"ALL"` | no |
| <a name="input_igw_tags"></a> [igw\_tags](#input\_igw\_tags) | Additional tags for the internet gateway | `map(string)` | `{}` | no |
| <a name="input_instance_tenancy"></a> [instance\_tenancy](#input\_instance\_tenancy) | A tenancy option for instances launched into the VPC | `string` | `"default"` | no |
......
examples/complete-vpc/README.md
View file @ 2423e4ff
......@@ -21,14 +21,14 @@ Note that this example may create resources which can cost money (AWS Elastic IP
| Name | Version |
|------|---------|
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.12.31 |
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.28 |
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.13.1 |
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.63 |
## Providers
| Name | Version |
|------|---------|
| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 3.28 |
| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 3.63 |
## Modules
......
examples/complete-vpc/versions.tf
View file @ 2423e4ff
terraform {
required_version = ">= 0.12.31"
required_version = ">= 0.13.1"
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 3.28"
version = ">= 3.63"
}
}
}
examples/ipv6/README.md
View file @ 2423e4ff
......@@ -19,8 +19,8 @@ Note that this example may create resources which can cost money (AWS Elastic IP
| Name | Version |
|------|---------|
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.12.31 |
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.28 |
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.13.1 |
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.63 |
## Providers
......
examples/ipv6/versions.tf
View file @ 2423e4ff
terraform {
required_version = ">= 0.12.31"
required_version = ">= 0.13.1"
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 3.28"
version = ">= 3.63"
}
}
}
examples/issues/README.md
View file @ 2423e4ff
......@@ -24,8 +24,8 @@ Note that this example may create resources which can cost money (AWS Elastic IP
| Name | Version |
|------|---------|
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.12.31 |
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.28 |
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.13.1 |
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.63 |
## Providers
......
examples/issues/versions.tf
View file @ 2423e4ff
terraform {
required_version = ">= 0.12.31"
required_version = ">= 0.13.1"
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 3.28"
version = ">= 3.63"
}
}
}
examples/manage-default-vpc/README.md
View file @ 2423e4ff
......@@ -21,8 +21,8 @@ Run `terraform destroy` when you don't need these resources.
| Name | Version |
|------|---------|
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.12.31 |
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.28 |
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.13.1 |
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.63 |
## Providers
......
examples/manage-default-vpc/versions.tf
View file @ 2423e4ff
terraform {
required_version = ">= 0.12.31"
required_version = ">= 0.13.1"
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 3.28"
version = ">= 3.63"
}
}
}
examples/network-acls/README.md
View file @ 2423e4ff
......@@ -23,8 +23,8 @@ Note that this example may create resources which can cost money (AWS Elastic IP
| Name | Version |
|------|---------|
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.12.31 |
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.28 |
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.13.1 |
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.63 |
## Providers
......
examples/network-acls/versions.tf
View file @ 2423e4ff
terraform {
required_version = ">= 0.12.31"
required_version = ">= 0.13.1"
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 3.28"
version = ">= 3.63"
}
}
}
examples/outpost/README.md
View file @ 2423e4ff
......@@ -23,14 +23,14 @@ Note that this example may create resources which can cost money (AWS Elastic IP
| Name | Version |
|------|---------|
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.12.31 |
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.28 |
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.13.1 |
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.63 |
## Providers
| Name | Version |
|------|---------|
| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 3.28 |
| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 3.63 |
## Modules
......
examples/outpost/versions.tf
View file @ 2423e4ff
terraform {
required_version = ">= 0.12.31"
required_version = ">= 0.13.1"
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 3.28"
version = ">= 3.63"
}
}
}
examples/secondary-cidr-blocks/README.md
View file @ 2423e4ff
......@@ -21,8 +21,8 @@ Note that this example may create resources which can cost money (AWS Elastic IP
| Name | Version |
|------|---------|
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.12.31 |
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.28 |
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.13.1 |
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.63 |
## Providers
......
examples/secondary-cidr-blocks/versions.tf
View file @ 2423e4ff
terraform {
required_version = ">= 0.12.31"
required_version = ">= 0.13.1"
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 3.28"
version = ">= 3.63"
}
}
}
examples/simple-vpc/README.md
View file @ 2423e4ff
......@@ -25,8 +25,8 @@ Note that this example may create resources which can cost money (AWS Elastic IP
| Name | Version |
|------|---------|
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.12.31 |
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.28 |
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.13.1 |
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.63 |
## Providers
......
examples/simple-vpc/versions.tf
View file @ 2423e4ff
terraform {
required_version = ">= 0.12.31"
required_version = ">= 0.13.1"
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 3.28"
version = ">= 3.63"
}
}
}
examples/vpc-flow-logs/README.md
View file @ 2423e4ff
......@@ -23,15 +23,15 @@ Note that this example may create resources which can cost money (AWS Elastic IP
| Name | Version |
|------|---------|
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.12.31 |
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.28 |
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.13.1 |
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.63 |
| <a name="requirement_random"></a> [random](#requirement\_random) | >= 2 |
## Providers
| Name | Version |
|------|---------|
| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 3.28 |
| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 3.63 |
| <a name="provider_random"></a> [random](#provider\_random) | >= 2 |
## Modules
......@@ -42,6 +42,7 @@ Note that this example may create resources which can cost money (AWS Elastic IP
| <a name="module_vpc_with_flow_logs_cloudwatch_logs"></a> [vpc\_with\_flow\_logs\_cloudwatch\_logs](#module\_vpc\_with\_flow\_logs\_cloudwatch\_logs) | ../../ | |
| <a name="module_vpc_with_flow_logs_cloudwatch_logs_default"></a> [vpc\_with\_flow\_logs\_cloudwatch\_logs\_default](#module\_vpc\_with\_flow\_logs\_cloudwatch\_logs\_default) | ../../ | |
| <a name="module_vpc_with_flow_logs_s3_bucket"></a> [vpc\_with\_flow\_logs\_s3\_bucket](#module\_vpc\_with\_flow\_logs\_s3\_bucket) | ../../ | |
| <a name="module_vpc_with_flow_logs_s3_bucket_parquet"></a> [vpc\_with\_flow\_logs\_s3\_bucket\_parquet](#module\_vpc\_with\_flow\_logs\_s3\_bucket\_parquet) | ../../ | |
## Resources
......
examples/vpc-flow-logs/main.tf
View file @ 2423e4ff
......@@ -31,6 +31,25 @@ module "vpc_with_flow_logs_s3_bucket" {
}
}
module "vpc_with_flow_logs_s3_bucket_parquet" {
source = "../../"
name = "vpc-flow-logs-s3-bucket"
cidr = "10.30.0.0/16"
azs = ["${local.region}a"]
public_subnets = ["10.30.101.0/24"]
enable_flow_log = true
flow_log_destination_type = "s3"
flow_log_destination_arn = module.s3_bucket.this_s3_bucket_arn
flow_log_file_format = "parquet"
vpc_flow_log_tags = {
Name = "vpc-flow-logs-s3-bucket"
}
}
# CloudWatch Log Group and IAM role created automatically
module "vpc_with_flow_logs_cloudwatch_logs_default" {
source = "../../"
......
examples/vpc-flow-logs/versions.tf
View file @ 2423e4ff
terraform {
required_version = ">= 0.12.31"
required_version = ">= 0.13.1"
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 3.28"
version = ">= 3.63"
}
random = {
......
examples/vpc-separate-private-route-tables/README.md
View file @ 2423e4ff
......@@ -21,8 +21,8 @@ Note that this example may create resources which can cost money (AWS Elastic IP
| Name | Version |
|------|---------|
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.12.31 |
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.28 |
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.13.1 |
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.63 |
## Providers
......
examples/vpc-separate-private-route-tables/versions.tf
View file @ 2423e4ff
terraform {
required_version = ">= 0.12.31"
required_version = ">= 0.13.1"
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 3.28"
version = ">= 3.63"
}
}
}
modules/vpc-endpoints/README.md
View file @ 2423e4ff
......@@ -55,7 +55,7 @@ module "endpoints" {
| Name | Version |
|------|---------|
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.12.31 |
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.13.1 |
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.28 |
## Providers
......
modules/vpc-endpoints/versions.tf
View file @ 2423e4ff
terraform {
required_version = ">= 0.12.31"
required_version = ">= 0.13.1"
required_providers {
aws = {
......
variables.tf
View file @ 2423e4ff
......@@ -1151,3 +1151,26 @@ variable "outpost_az" {
type = string
default = null
}
variable "flow_log_file_format" {
description = "(Optional) The format for the flow log. Valid values: `plain-text`, `parquet`."
type = string
default = "plain-text"
validation {
condition = can(regex("^(plain-text|parquet)$",
var.flow_log_file_format))
error_message = "ERROR valid values: plain-text, parquet."
}
}
variable "flow_log_hive_compatible_partitions" {
description = "(Optional) Indicates whether to use Hive-compatible prefixes for flow logs stored in Amazon S3."
type = bool
default = false
}
variable "flow_log_per_hour_partition" {
description = "(Optional) Indicates whether to partition the flow log per hour. This reduces the cost and response time for queries."
type = bool
default = false
}
versions.tf
View file @ 2423e4ff
terraform {
required_version = ">= 0.12.31"
required_version = ">= 0.13.1"
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 3.38"
version = ">= 3.63"
}
}
}
vpc-flow-logs.tf
View file @ 2423e4ff
......@@ -24,6 +24,12 @@ resource "aws_flow_log" "this" {
vpc_id = local.vpc_id
max_aggregation_interval = var.flow_log_max_aggregation_interval
destination_options {
file_format = var.flow_log_file_format
hive_compatible_partitions = var.flow_log_hive_compatible_partitions
per_hour_partition = var.flow_log_per_hour_partition
}
tags = merge(var.tags, var.vpc_flow_log_tags)
}
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment