terravision

Resource Handler Guide

Table of Contents

  1. Overview
  2. Critical: Validate Baseline First
  3. Handler Architecture
  4. Configuration Structure
  5. Handler Types in Detail
  6. Available Transformers
  7. Resource Consolidation
  8. Decision Guide
  9. Migration Examples
  10. Architecture Components
  11. Execution Flow
  12. Adding New Resource Types
  13. Testing Strategy
  14. Performance Considerations
  15. Best Practices
  16. Current State (AWS)

Overview

TerraVision uses a unified hybrid configuration-driven approach for all resource handlers. This architecture allows handlers to be implemented in three ways:

  1. Pure config-driven: Use only declarative transformation building blocks (70% code reduction)
  2. Hybrid: Use transformations + custom Python function (best of both worlds)
  3. Pure function: Use only custom Python function (for complex logic)

All handlers are defined in modules/config/resource_handler_configs_<provider>.py with automatic provider detection.

Problem Statement

The original resource_handlers_aws.py contained 30+ handler functions with repetitive patterns:

This led to:

Solution

The hybrid handler architecture provides:

Critical: Validate Baseline First

⚠️ IMPORTANT: Before implementing ANY handler, you MUST prove that the baseline Terraform graph parsing is insufficient.

Validation Process

Step 1: Generate Baseline

# Create test Terraform code for the resource type
cd tests/fixtures/aws_terraform/test_resource/

# Run TerraVision WITHOUT any custom handler
# IMPORTANT: Use --debug to save tfdata.json for test reuse
poetry run python ../../../terravision.py graphdata --source . --outfile baseline.json --debug

# The --debug flag creates tfdata.json which can be reused:
poetry run python ../../../terravision.py graphdata --source tfdata.json --outfile output.json

Step 2: Analyze Baseline Output

Step 3: Decision

Real Example: API Gateway

Baseline Output (no handler):

Lambda → Integration → Method → Resource → REST API

Analysis:

Decision: ❌ Handler NOT needed - baseline is sufficient!

Mistake Made: Tried to parse integration URIs to create “better” diagrams
Result: Added complexity, created unhelpful placeholder nodes
Lesson: Baseline was already good - handler made it worse!

Counter-Example: Security Groups

Baseline Output (no handler):

EC2 → Security Group (simple dependency arrow)

Analysis:

Decision: ✅ Handler needed - baseline insufficient!

Implementation: Parse SG rules, create directional arrows with port/protocol labels

Handler Architecture

Three Handler Types

1. Pure Config-Driven (7 AWS handlers)

Use when: Simple, repetitive operations (expand, move, link, delete)

Benefits:

Example:

"aws_vpc_endpoint": {
    "transformations": [
        {"operation": "move_to_parent", "params": {...}},
        {"operation": "delete_nodes", "params": {...}},
    ],
}

Current AWS handlers:

2. Hybrid (3 AWS handlers)

Use when: Common operations + unique logic

Benefits:

Example:

"aws_subnet": {
    "handler_execution_order": "before",  # Run custom function FIRST
    "additional_handler_function": "aws_prepare_subnet_az_metadata",
    "transformations": [
        {"operation": "insert_intermediate_node", "params": {...}},
    ],
}

Current AWS handlers:

3. Pure Function (6 AWS handlers)

Use when: Complex logic that can’t be expressed declaratively

Reasons for complexity:

Example:

"aws_security_group": {
    "additional_handler_function": "aws_handle_sg",
}

Current AWS specific handlers:

Why Hybrid Architecture?

Problem: Some handlers are simple (move, link, delete), others are complex (conditional logic, dynamic naming, metadata manipulation).

Solution: Use the right tool for each handler:

Configuration Structure

Basic Structure

RESOURCE_HANDLER_CONFIGS = {
    "resource_pattern": {
        "description": "What this handler does",
        "transformations": [  # Optional: config-driven transformations
            {
                "operation": "transformer_name",
                "params": {
                    "param1": "value1",
                    "param2": "value2",
                }
            },
        ],
        "additional_handler_function": "function_name",  # Optional: custom Python function
        "handler_execution_order": "after",  # Optional: "before" or "after" (default)
    },
}

Execution Order

By default, when a resource pattern matches:

  1. Config-driven transformations are applied first (if transformations key exists)
  2. Additional handler function is called second (if additional_handler_function key exists)

You can reverse this order using the handler_execution_order parameter:

"handler_execution_order": "before",  # Run custom function BEFORE transformations
"handler_execution_order": "after",   # Run custom function AFTER transformations (default)

Use Case for “before”: When transformations need prepared metadata from the custom function.

Example:

"aws_subnet": {
    "handler_execution_order": "before",  # Run metadata prep FIRST
    "additional_handler_function": "aws_prepare_subnet_az_metadata",  # Copies metadata
    "transformations": [
        {"operation": "insert_intermediate_node", ...},  # Uses prepared metadata
    ],
}

Automatic Function Resolution

Parameters ending in _function or _generator are automatically resolved from string names to actual function references from handler modules (resource_handlers_aws.py, resource_handlers_gcp.py, resource_handlers_azure.py).

Example:

"transformations": [
    {
        "operation": "insert_intermediate_node",
        "params": {
            "intermediate_node_generator": "generate_az_node_name",  # String in config
        },
    },
]
# Automatically resolved at runtime to: handlers_aws.generate_az_node_name (function reference)

This allows configuration files to specify callable parameters without module imports or manual function mapping.

Handler Types in Detail

1. Pure Config-Driven Handler

Uses only transformation building blocks. No custom Python code needed.

Example: VPC Endpoint Handler

"aws_vpc_endpoint": {
    "description": "Move VPC endpoints into VPC parent and delete endpoint nodes",
    "transformations": [
        {
            "operation": "move_to_parent",
            "params": {
                "resource_pattern": "aws_vpc_endpoint",
                "from_parent_pattern": "aws_subnet",
                "to_parent_pattern": "aws_vpc.",
            },
        },
        {
            "operation": "delete_nodes",
            "params": {
                "resource_pattern": "aws_vpc_endpoint",
                "remove_from_parents": False,
            },
        },
    ],
}

Result: 30 lines of Python → 10 lines of config (67% reduction)

2. Hybrid Handler

Uses transformations for common operations, then custom function for complex logic.

Example: Subnet Handler

"aws_subnet": {
    "description": "Create availability zone nodes and link to subnets",
    "handler_execution_order": "before",  # Run custom function FIRST
    "additional_handler_function": "aws_prepare_subnet_az_metadata",
    "transformations": [
        {
            "operation": "unlink_from_parents",
            "params": {
                "resource_pattern": "aws_subnet",
                "parent_filter": "aws_vpc",
            },
        },
        {
            "operation": "insert_intermediate_node",
            "params": {
                "parent_pattern": "aws_vpc",
                "child_pattern": "aws_subnet",
                "intermediate_node_generator": "generate_az_node_name",
                "create_if_missing": True,
            },
        },
    ],
}

Custom function (in resource_handlers_aws.py):

def aws_prepare_subnet_az_metadata(tfdata: Dict[str, Any]) -> Dict[str, Any]:
    """Prepare metadata for transformer to use."""
    for subnet in subnets:
        tfdata["meta_data"][subnet]["availability_zone"] = original["availability_zone"]
        tfdata["meta_data"][subnet]["region"] = original["region"]
    return tfdata

Result: 51 lines → 39 lines (24% reduction)

3. Pure Function Handler

Uses only custom Python function for complex logic that can’t be expressed with transformers.

Example: Security Group Handler

"aws_security_group": {
    "description": "Process security group relationships and reverse connections",
    "additional_handler_function": "aws_handle_sg",
}

Why pure function: Security groups require:

Available Transformers

All transformers are defined in modules/resource_transformers.py. Total: 23 generic transformers + 1 pipeline orchestrator.

Resource Expansion (2 transformers)

Resource Grouping (4 transformers)

⚠️ For Resource Consolidation: Use AWS_CONSOLIDATED_NODES in cloud_config_aws.py instead of transformers. See “Resource Consolidation” section below.

Connections (11 transformers)

Graph Manipulation (1 transformer)

Metadata Operations (1 transformer)

Cleanup (1 transformer)

Variants (2 transformers)

Pipeline (1 orchestrator)

Resource Consolidation

⚠️ IMPORTANT: When you need to consolidate multiple resources into a single node (e.g., merge all aws_api_gateway_rest_api.* into one aws_api_gateway_rest_api.api node), DO NOT use a transformer. Instead, add an entry to AWS_CONSOLIDATED_NODES in cloud_config_aws.py.

Why Use AWS_CONSOLIDATED_NODES?

  1. Simpler: Declarative configuration in one central location
  2. More maintainable: All consolidation rules in one place
  3. Existing mechanism: Uses the proven consolidation system already in TerraVision
  4. Automatic: Runs before handlers, so handlers see consolidated nodes

How to Consolidate Resources

❌ WRONG - Don’t use transformers:

# DON'T DO THIS!
"aws_api_gateway_rest_api": {
    "transformations": [
        {
            "operation": "consolidate_into_single_node",  # This transformer was removed!
            "params": {
                "resource_pattern": "aws_api_gateway_rest_api",
                "target_node_name": "aws_api_gateway_rest_api.api",
            },
        },
    ],
}

✅ CORRECT - Add to AWS_CONSOLIDATED_NODES:

# In modules/config/cloud_config_aws.py
AWS_CONSOLIDATED_NODES = [
    # ... existing entries ...
    {
        "aws_api_gateway_rest_api": {
            "resource_name": "aws_api_gateway_rest_api.api",
            "import_location": "resource_classes.aws.network",
            "vpc": False,
            "edge_service": True,
        }
    },
]

AWS_CONSOLIDATED_NODES Format

{
    "resource_prefix": {  # Pattern to match (e.g., "aws_api_gateway_rest_api")
        "resource_name": "consolidated.node.name",  # Final consolidated name
        "import_location": "resource_classes.aws.category",  # Icon path
        "vpc": True/False,  # Whether resource is inside VPC
        "edge_service": True/False,  # (Optional) Whether it's an edge service
    }
}

When to Use Consolidation

Use AWS_CONSOLIDATED_NODES when:

Note: After consolidation, you can still use handler transformations to add connections, delete sub-resources, etc.

Decision Guide

Decision Tree

Use this decision tree to determine the best approach for a new or existing handler:

Does the handler involve complex conditional logic?
├─ YES → Is the logic domain-specific (can't be generalized)?
│  ├─ YES → Use Pure Function
│  │         Examples: Security group reverse relationships,
│  │                   autoscaling count propagation with policy filtering
│  └─ NO → Can you break it into generic operations + custom logic?
│     ├─ YES → Use Hybrid
│     │         Examples: Subnet (metadata prep + insert intermediate node),
│     │                   EFS (bidirectional link + custom cleanup)
│     └─ NO → Use Pure Config-Driven
│               Examples: VPC endpoint (move + delete),
│                         EKS node group (expand to numbered instances)
└─ NO → Can the entire operation be expressed with existing transformers?
   ├─ YES → Use Pure Config-Driven
   │         Examples: DB subnet group (move + redirect),
   │                   Random string (delete nodes)
   └─ NO → Would creating a new generic transformer be useful?
      ├─ YES → Create transformer, then use Pure Config-Driven
      │         Examples: insert_intermediate_node, propagate_metadata
      └─ NO → Use Hybrid or Pure Function
                Examples: Custom domain parsing, chart-specific logic

Quick Reference Table

Characteristic Pure Config Hybrid Pure Function
Code lines 0 Python, 10-20 config 20-40 Python, 10-20 config 50-150 Python, 5 config
Complexity Simple operations Medium complexity High complexity
Reusability 100% (all generic) 50-80% (transformers reusable) 0% (all custom)
Testability Test transformers once Test function + transformers Test entire function
Maintainability Easiest (declarative) Medium (clear separation) Hardest (procedural)
Use When Simple, repetitive ops Common ops + unique logic Complex conditional logic

Red Flags and Green Lights

Red Flags for Pure Config - Avoid Pure Config if handler has ANY of these:

Green Lights for Pure Config - Use Pure Config if handler does ALL operations with existing transformers:

When to Choose Hybrid:

When to Keep as Pure Function:

Migration Examples

Example 1: Pure Config-Driven Handler

Before (Python function):

def aws_handle_vpc_endpoint(tfdata: Dict[str, Any]) -> Dict[str, Any]:
    # Move VPC endpoints from subnets to VPC parent
    vpc_endpoints = helpers.list_of_dictkeys_containing(
        tfdata["graphdict"], "aws_vpc_endpoint"
    )
    for endpoint in vpc_endpoints:
        # ... 30 lines of logic to move and delete
    return tfdata

After (Configuration):

"aws_vpc_endpoint": {
    "description": "Move VPC endpoints into VPC parent and delete endpoint nodes",
    "transformations": [
        {
            "operation": "move_to_parent",
            "params": {
                "resource_pattern": "aws_vpc_endpoint",
                "from_parent_pattern": "aws_subnet",
                "to_parent_pattern": "aws_vpc.",
            },
        },
        {
            "operation": "delete_nodes",
            "params": {
                "resource_pattern": "aws_vpc_endpoint",
                "remove_from_parents": False,
            },
        },
    ],
}

Result: 30 lines of Python → 10 lines of config (67% reduction)

Example 2: Hybrid Handler

Concept: Use config for common operations, custom function for complex logic

"aws_efs_file_system": {
    "description": "Handle EFS mount targets and file system relationships",
    "transformations": [
        {
            "operation": "bidirectional_link",
            "params": {
                "source_pattern": "aws_efs_mount_target",
                "target_pattern": "aws_efs_file_system",
                "cleanup_reverse": True,
            },
        },
    ],
    "additional_handler_function": "aws_handle_efs",  # Custom cleanup logic
}

Custom function (in resource_handlers_aws.py):

def aws_handle_efs(tfdata: Dict[str, Any]) -> Dict[str, Any]:
    """Handle complex logic that can't be expressed with transformers."""
    # Config already created bidirectional links
    # Now apply complex conditional cleanup logic
    ...
    return tfdata

Example 3: Pure Function Handler

When to use: Complex logic that can’t be expressed with transformers

"aws_security_group": {
    "description": "Process security group relationships and reverse connections",
    "additional_handler_function": "aws_handle_sg",
}

Why pure function: Security groups require:

Architecture Components

Configuration Files

Transformer Functions

Handler Functions

Execution Engine

Execution Flow

# In graphmaker.py
def handle_special_resources(tfdata: Dict[str, Any]) -> Dict[str, Any]:
    """Execute all configured handlers in sequence."""
    
    # Load provider-specific handler configs (automatic detection)
    handler_configs = _load_handler_configs(tfdata)
    
    for resource_pattern, config in handler_configs.items():
        # Check if this resource type exists in graph
        if helpers.list_of_dictkeys_containing(tfdata["graphdict"], resource_pattern):
            
            # Determine execution order
            order = config.get("handler_execution_order", "after")
            
            if order == "before":
                # Step 1: Call custom handler function first (if present)
                if "additional_handler_function" in config:
                    handler_func = getattr(resource_handlers, config["additional_handler_function"])
                    tfdata = handler_func(tfdata)
                
                # Step 2: Apply config-driven transformations (if present)
                if "transformations" in config:
                    tfdata = apply_transformation_pipeline(
                        tfdata, 
                        config["transformations"]
                    )
            else:  # "after" (default)
                # Step 1: Apply config-driven transformations (if present)
                if "transformations" in config:
                    tfdata = apply_transformation_pipeline(
                        tfdata, 
                        config["transformations"]
                    )
                
                # Step 2: Call custom handler function (if present)
                if "additional_handler_function" in config:
                    handler_func = getattr(resource_handlers, config["additional_handler_function"])
                    tfdata = handler_func(tfdata)
    
    return tfdata

Execution Steps:

  1. Load provider-specific handler configs (automatic detection)
  2. For each resource pattern in config:
    • Check if resource exists in graph
    • If handler_execution_order is “before”:
      • Call custom function first
      • Apply transformations second
    • If handler_execution_order is “after” (default):
      • Apply transformations first
      • Call custom function second
  3. Continue to next handler

Adding New Resource Types

Step 1: Identify Pattern

Determine which approach is needed:

Step 2: Add Configuration

Add entry to RESOURCE_HANDLER_CONFIGS in modules/config/resource_handler_configs_<provider>.py:

Pure config example:

"aws_new_resource": {
    "description": "Handle new resource type",
    "transformations": [
        {
            "operation": "expand_to_numbered_instances",
            "params": {
                "resource_pattern": "aws_new_resource",
                "subnet_key": "subnet_ids",
            },
        },
    ],
}

Hybrid example:

"aws_new_resource": {
    "description": "Handle new resource type",
    "transformations": [
        {"operation": "expand_to_numbered_instances", "params": {...}},
    ],
    "additional_handler_function": "aws_handle_new_resource_custom",
}

Pure function example:

"aws_new_resource": {
    "description": "Handle new resource type with complex logic",
    "additional_handler_function": "aws_handle_new_resource",
}

Step 3: Test

Run TerraVision with Terraform code containing the new resource type:

cd tests/fixtures/aws_terraform/test_new_resource/
poetry run python ../../../terravision.py draw --source . --debug

Testing Strategy

Unit Tests

Test each transformer independently:

def test_expand_to_numbered_instances():
    tfdata = {
        "graphdict": {
            "aws_eks_node_group.workers": [],
            "aws_subnet.private_1": [],
            "aws_subnet.private_2": [],
        },
        "meta_data": {
            "aws_eks_node_group.workers": {
                "subnet_ids": ["subnet-1", "subnet-2"]
            },
            "aws_subnet.private_1": {"id": "subnet-1"},
            "aws_subnet.private_2": {"id": "subnet-2"},
        },
    }
    
    result = expand_to_numbered_instances(
        tfdata, 
        "aws_eks_node_group",
        "subnet_ids"
    )
    
    assert "aws_eks_node_group.workers~1" in result["graphdict"]
    assert "aws_eks_node_group.workers~2" in result["graphdict"]

Integration Tests

Test complete handler configurations:

def test_eks_node_group_handler():
    tfdata = load_test_data("eks_cluster.json")
    config = AWS_RESOURCE_HANDLER_CONFIGS["aws_eks_node_group"]
    
    result = apply_transformation_pipeline(tfdata, config["transformations"])
    
    # Verify expected output
    assert_node_groups_expanded(result)
    assert_linked_to_subnets(result)

Regression Tests

Compare output with original handler functions:

def test_backward_compatibility():
    tfdata = load_test_data("complex_architecture.json")
    
    # Old approach
    old_result = match_node_groups_to_subnets(tfdata.copy())
    
    # New approach
    new_result = apply_transformation_pipeline(
        tfdata.copy(),
        AWS_RESOURCE_HANDLER_CONFIGS["aws_eks_node_group"]["transformations"]
    )
    
    assert old_result == new_result

Validation Checklist

Before declaring a handler complete, verify:

Performance Considerations

Optimization Opportunities

  1. Lazy Evaluation - Only execute handlers for resources present in graph
  2. Caching - Cache pattern matching results
  3. Parallel Execution - Run independent transformations in parallel
  4. Early Exit - Skip transformations if preconditions not met

Benchmarking

import time

def benchmark_handler(handler_name, tfdata):
    start = time.time()
    config = AWS_RESOURCE_HANDLER_CONFIGS[handler_name]
    result = apply_transformation_pipeline(tfdata, config["transformations"])
    elapsed = time.time() - start
    print(f"{handler_name}: {elapsed:.3f}s")
    return result

Best Practices

  1. Start with Pure Config: Always try Pure Config first. If you can’t express the logic with existing transformers, consider Hybrid or Pure Function.

  2. Create Transformers for Common Patterns: If you see a pattern used in multiple handlers, create a generic transformer.

  3. Keep Functions Small: If custom function is >100 lines, consider breaking it into multiple transformers or helper functions.

  4. Document Handler Type: Always include "description" explaining what the handler does and why it’s that type.

  5. Test Behavior Preservation: When refactoring, ensure tests pass without changing expected output.

  6. Use handler_execution_order: When transformers need prepared data, run custom function first with "handler_execution_order": "before".

  7. Avoid Over-Engineering: Don’t create transformers for one-off operations. Pure Function is fine for unique, complex logic.

  8. Complete Validation Checklist: Before declaring a handler complete, run the comprehensive validation checklist to catch:
    • Connection direction errors (arrows pointing wrong way)
    • Orphaned resources (missing connections)
    • Duplicate connections
    • Intermediary link issues (transitive links not created)
    • Test coverage gaps

  9. Trust the Baseline: Always validate that a handler is needed before implementing. The baseline Terraform graph parsing may already be sufficient.

  10. Use Consolidation Correctly: For merging multiple resources into one node, use AWS_CONSOLIDATED_NODES in cloud_config_aws.py, not transformers.

Current State (AWS)

Pure Config-Driven (7 handlers)

Code Reduction: 360 lines → 85 lines (70% reduction)

Hybrid (3 handlers)

Pure Function (6 handlers - too complex for config)

Benefits Summary

1. Maintainability

2. Consistency

3. Extensibility

4. Documentation

5. Code Reduction

6. Flexibility

Future Enhancements

Conditional Transformations

{
    "operation": "expand_to_numbered_instances",
    "condition": {
        "metadata_key": "multi_az",
        "equals": True,
    },
    "params": {...},
}

Parameterized Transformations

{
    "operation": "create_group_node",
    "params": {
        "group_name": "aws_account.{cluster_name}_control_plane",
        "children": ["aws_eks_cluster.{cluster_name}"],
    },
}

Validation Rules

{
    "operation": "link_resources",
    "validate": {
        "source_exists": True,
        "target_exists": True,
    },
    "params": {...},
}

Conclusion

The hybrid configuration-driven approach reduces code by ~70% for simple handlers while maintaining flexibility for complex logic:

This architecture makes TerraVision easier to extend with new cloud services while keeping complex logic maintainable. All 135 tests pass with the hybrid architecture, proving backward compatibility and correctness.

This site is open source. Improve this page.