How to Export and Import Pipelines

This guide explains how to export and import pipelines with enhanced functionality that includes queue configurations and resolves ID conflicts.

Overview

The pipeline export/import system has been improved to address two main issues:

1. ID Conflicts: Previously, exported pipelines maintained the same node IDs, causing conflicts when imported into new pipelines
2. Missing Queue Configuration: Queue configurations for manual nodes were not included in exports

Enhanced Export Features

What's Included in Exports

The new export format includes:

• Flow Structure: All nodes and edges with unique IDs
• Queue Configurations: Complete queue settings for manual nodes
• Metadata: Pipeline information and export details

Export Format

{
  "flow": {
    "nodes": [...],
    "edges": [...]
  },
  "queues": [...],
  "metadata": {
    "name": "Pipeline Name",
    "description": "Pipeline Description",
    "exportedAt": "2024-01-01T00:00:00.000Z",
    "originalPipelineId": "original-pipeline-id"
  }
}

How to Export a Pipeline

Step 1: Access Export Function

• Navigate to your pipeline in the editor
• Click the "More" button (⋮) in the top toolbar
• Select "Export" from the dropdown menu

Step 2: Review Export Data

• A modal will open showing the complete export data
• The JSON includes:
  • Flow structure with unique node IDs
  • Queue configurations for all manual nodes
  • Metadata with pipeline information

Step 3: Copy Export Data

• Click "Export pipeline" to copy the JSON to clipboard
• Or manually select and copy the JSON content
• Save the export data for later use

How to Import a Pipeline

Step 1: Access Import Function

• Open a new or existing pipeline
• Click the "More" button (⋮) in the top toolbar
• Select "Import" from the dropdown menu

Step 2: Paste Export Data

• In the import modal, paste the JSON export data
• The system will validate the format automatically

Step 3: Import Pipeline

• Click "Import pipeline" to apply the import
• The system will:
  • Generate new unique IDs for all nodes
  • Update all edge references
  • Create queue configurations
  • Maintain all relationships

ID Generation System

Unique ID Format

Each exported node receives a unique ID using this format:

node_${timestamp}_${randomString}

Example: node_1704067200000_a1b2c3d

ID Mapping

The system maintains a mapping of old IDs to new IDs:

const idMapping: Record<string, string> = {
  'old-node-id': 'new-node-id',
  'another-old-id': 'another-new-id',
};

Benefits

• No Conflicts: Each import generates completely unique IDs
• Consistent References: All edges and queue references are updated
• Event Compatibility: Events tied to specific node IDs work correctly

Queue Configuration Export

What's Exported

Queue configurations include:

• Basic Information: Name, description, status
• Settings: Mode (resolve/review), filters, instructions
• Node References: Updated to match new node IDs
• Configuration: All queue-specific settings

What's Not Exported

The following Firestore-specific fields are excluded:

• pipelineRef: Will be set to the new pipeline
• createdAt: Will be set to current timestamp
• createdBy: Will be set to current user
• lastUpdatedAt: Will be set to current timestamp
• lastUpdatedBy: Will be set to current user

Import Process Details

Validation

The import process validates:

• Flow Structure: Ensures nodes and edges are properly formatted
• Queue Data: Validates queue configurations if present
• Metadata: Checks for required pipeline information

ID Generation

During import:

1. New Node IDs: Each node gets a completely new unique ID
2. Edge Updates: All edge source/target references are updated
3. Queue Updates: Queue nodeId references are updated
4. Relationship Maintenance: All connections remain intact

Queue Creation

If queues are included in the import:

• Automatic Creation: Queues are created in the new pipeline
• Updated References: nodeId references are updated to new IDs
• Fresh Metadata: Creation timestamps and user references are set

Best Practices

For Exporting

• Export Regularly: Export pipelines before major changes
• Test Imports: Import into a test pipeline to verify functionality
• Backup Exports: Keep multiple export versions for important pipelines

For Importing

• Review Data: Check the import preview before applying
• Test Functionality: Verify that all nodes and queues work correctly
• Update References: Update any external references to the old pipeline

General Tips

• Unique Names: Use descriptive names for exported pipelines
• Version Control: Keep track of different export versions
• Documentation: Document any custom configurations or dependencies

Troubleshooting

Common Issues

"Invalid import format" error:

• Ensure the JSON is properly formatted
• Check that the export data is complete
• Verify the JSON structure matches the expected format

Missing queues after import:

• Check that the original pipeline had queue configurations
• Verify that the export included queue data
• Ensure the import process completed successfully

Node ID conflicts:

• This should no longer occur with the new system
• If conflicts persist, check for manual ID modifications

Queue not working:

• Verify that queue nodeId references were updated
• Check that the queue was created in the new pipeline
• Ensure the node type supports queues (manual nodes)

Getting Help

If you encounter issues:

1. Check Export Data: Verify the JSON structure is correct
2. Test Import: Try importing into a test pipeline first
3. Review Logs: Check browser console for error messages
4. Contact Support: Reach out to your system administrator

Technical Details

Export Process

1. Flow Extraction: Gets current flow state from React Flow
2. Queue Collection: Fetches all queues for the pipeline
3. ID Generation: Creates new unique IDs for all nodes
4. Reference Updates: Updates all edge and queue references
5. Data Cleaning: Removes Firestore-specific fields
6. JSON Generation: Creates the final export format

Import Process

1. Data Validation: Validates the import JSON structure
2. ID Generation: Creates new unique IDs for imported nodes
3. Reference Updates: Updates all edge and queue references
4. Flow Application: Applies the updated flow to React Flow
5. Queue Creation: Creates queues in the new pipeline
6. Error Handling: Manages any import errors gracefully

Summary

The enhanced export/import system provides:

• Conflict-Free IDs: No more ID conflicts between pipelines
• Complete Queue Support: Full queue configuration export/import
• Robust Validation: Better error handling and data validation
• Backward Compatibility: Works with existing pipeline formats

This system ensures that pipeline exports are complete, reliable, and ready for use in new environments without conflicts or missing configurations.

---

Related Guides

• How to Manage Queue Users
• How to Create a Pipeline
• How to Add Members