Home Explore Help
Sign In
mirrors
/
wekan
mirror of https://github.com/wekan/wekan.git
2
0
Fork 0
Files Issues 0 Wiki
Tree: 7f31d7c812
Branches Tags
wekan
/
docs
/
ImportExport
/
Attachment-Migration-System.md

Attachment-Migration-System.md 10 KB
History Raw

Enhanced Attachment Migration System

Overview

The Enhanced Attachment Migration System provides a comprehensive solution for managing attachment storage across multiple backends (filesystem, MongoDB GridFS, S3/MinIO) with CPU throttling, real-time monitoring, and secure configuration management.

Features

1. Multi-Backend Storage Support

  • Filesystem Storage: Local file system storage using WRITABLE_PATH
  • MongoDB GridFS: Database-based file storage
  • S3/MinIO: Cloud and object storage compatibility

2. CPU Throttling

  • Automatic CPU Monitoring: Real-time CPU usage tracking
  • Configurable Thresholds: Set CPU usage limits (10-90%)
  • Automatic Pausing: Migration pauses when CPU threshold is exceeded
  • Resume Capability: Continue migration when CPU usage drops

3. Batch Processing

  • Configurable Batch Size: Process 1-100 attachments per batch
  • Adjustable Delays: Set delays between batches (100-10000ms)
  • Progress Tracking: Real-time progress monitoring
  • Queue Management: Intelligent migration queue handling

4. Security Features

  • Password Protection: S3 secret keys are never displayed
  • Admin-Only Access: All operations require admin privileges
  • Secure Configuration: Environment-based configuration management
  • Audit Logging: Comprehensive migration logging

5. Real-Time Monitoring

  • Storage Statistics: Live attachment counts and sizes
  • Visual Charts: Storage distribution visualization
  • Migration Status: Real-time migration progress
  • System Metrics: CPU, memory, and performance monitoring

Admin Panel Interface

Storage Settings

  • Filesystem Configuration: View and configure filesystem paths
  • GridFS Status: Monitor MongoDB GridFS availability
  • S3/MinIO Configuration: Secure S3/MinIO setup and testing
  • Connection Testing: Validate storage backend connections

Migration Controls

  • Batch Configuration: Set batch size, delays, and CPU thresholds
  • Migration Actions: Start, pause, resume, and stop migrations
  • Progress Monitoring: Real-time progress bars and statistics
  • Log Viewing: Live migration logs with timestamps

Monitoring Dashboard

  • Storage Distribution: Visual breakdown of attachment storage
  • Size Analytics: Total and per-storage size statistics
  • Performance Metrics: System resource usage
  • Export Capabilities: Download monitoring data

Configuration

Environment Variables

Filesystem Storage

# Base writable path for all file storage
WRITABLE_PATH=/data

# Attachments will be stored at: ${WRITABLE_PATH}/attachments
# Avatars will be stored at: ${WRITABLE_PATH}/avatars

S3/MinIO Storage

# S3 configuration (JSON format)
S3='{"s3":{"key":"access-key","secret":"secret-key","bucket":"bucket-name","endPoint":"s3.amazonaws.com","port":443,"sslEnabled":true,"region":"us-east-1"}}'

# Alternative: S3 secret file (Docker secrets)
S3_SECRET_FILE=/run/secrets/s3_secret

Migration Settings

Default Configuration

  • Batch Size: 10 attachments per batch
  • Delay: 1000ms between batches
  • CPU Threshold: 70% maximum CPU usage
  • Auto-pause: When CPU exceeds threshold

Customization

All settings can be adjusted through the admin panel:

  • Batch size: 1-100 attachments
  • Delay: 100-10000ms
  • CPU threshold: 10-90%

Usage

Starting a Migration

  1. Access Admin Panel: Navigate to Settings → Attachment Settings
  2. Configure Migration: Set batch size, delay, and CPU threshold
  3. Select Target Storage: Choose filesystem, GridFS, or S3
  4. Start Migration: Click the appropriate migration button
  5. Monitor Progress: Watch real-time progress and logs

Migration Process

  1. Queue Initialization: All attachments are queued for migration
  2. Batch Processing: Attachments are processed in configurable batches
  3. CPU Monitoring: System continuously monitors CPU usage
  4. Automatic Pausing: Migration pauses if CPU threshold is exceeded
  5. Resume Capability: Migration resumes when CPU usage drops
  6. Progress Tracking: Real-time progress updates and logging

Monitoring Migration

  • Progress Bar: Visual progress indicator
  • Statistics: Total, migrated, and remaining attachment counts
  • Status Display: Current migration status (running, paused, idle)
  • Log View: Real-time migration logs with timestamps
  • Control Buttons: Pause, resume, and stop migration

API Reference

Server Methods

Migration Control

// Start migration
Meteor.call('startAttachmentMigration', {
  targetStorage: 'filesystem', // 'filesystem', 'gridfs', 's3'
  batchSize: 10,
  delayMs: 1000,
  cpuThreshold: 70
});

// Pause migration
Meteor.call('pauseAttachmentMigration');

// Resume migration
Meteor.call('resumeAttachmentMigration');

// Stop migration
Meteor.call('stopAttachmentMigration');

Configuration Management

// Get storage configuration
Meteor.call('getAttachmentStorageConfiguration');

// Test S3 connection
Meteor.call('testS3Connection', { secretKey: 'new-secret-key' });

// Save S3 settings
Meteor.call('saveS3Settings', { secretKey: 'new-secret-key' });

Monitoring

// Get monitoring data
Meteor.call('getAttachmentMonitoringData');

// Refresh monitoring data
Meteor.call('refreshAttachmentMonitoringData');

// Export monitoring data
Meteor.call('exportAttachmentMonitoringData');

Publications

Real-Time Updates

// Subscribe to migration status
Meteor.subscribe('attachmentMigrationStatus');

// Subscribe to monitoring data
Meteor.subscribe('attachmentMonitoringData');

Performance Considerations

CPU Throttling

  • Automatic Monitoring: CPU usage is checked every 5 seconds
  • Threshold-Based Pausing: Migration pauses when CPU exceeds threshold
  • Resume Logic: Migration resumes when CPU usage drops below threshold
  • Configurable Limits: CPU thresholds can be adjusted (10-90%)

Batch Processing

  • Configurable Batches: Batch size can be adjusted (1-100)
  • Delay Control: Delays between batches prevent system overload
  • Queue Management: Intelligent queue processing with error handling
  • Progress Tracking: Real-time progress updates

Memory Management

  • Streaming Processing: Large files are processed using streams
  • Memory Monitoring: System memory usage is tracked
  • Garbage Collection: Automatic cleanup of processed data
  • Error Recovery: Robust error handling and recovery

Security

Access Control

  • Admin-Only Access: All operations require admin privileges
  • User Authentication: Proper user authentication and authorization
  • Session Management: Secure session handling

Data Protection

  • Password Security: S3 secret keys are never displayed in UI
  • Environment Variables: Sensitive data stored in environment variables
  • Secure Transmission: All data transmission is encrypted
  • Audit Logging: Comprehensive logging of all operations

Configuration Security

  • Read-Only Display: Sensitive configuration is displayed as read-only
  • Password Updates: Only new passwords can be set, not viewed
  • Connection Testing: Secure connection testing without exposing credentials
  • Environment Isolation: Configuration isolated from application code

Troubleshooting

Common Issues

Migration Not Starting

  • Check Permissions: Ensure user has admin privileges
  • Verify Configuration: Check storage backend configuration
  • Review Logs: Check server logs for error messages
  • Test Connections: Verify storage backend connectivity

High CPU Usage

  • Reduce Batch Size: Decrease batch size to reduce CPU load
  • Increase Delays: Add longer delays between batches
  • Lower CPU Threshold: Reduce CPU threshold for earlier pausing
  • Monitor System: Check system resource usage

Migration Pausing Frequently

  • Check CPU Threshold: Verify CPU threshold settings
  • Monitor System Load: Check for other high-CPU processes
  • Adjust Settings: Increase CPU threshold or reduce batch size
  • System Optimization: Optimize system performance

Storage Connection Issues

  • Verify Credentials: Check S3/MinIO credentials
  • Test Connectivity: Use connection test feature
  • Check Network: Verify network connectivity
  • Review Configuration: Validate storage configuration

Debug Information

Migration Logs

  • Real-Time Logs: View live migration logs in admin panel
  • Server Logs: Check server console for detailed logs
  • Error Messages: Review error messages for specific issues
  • Progress Tracking: Monitor migration progress and statistics

System Monitoring

  • CPU Usage: Monitor CPU usage during migration
  • Memory Usage: Track memory consumption
  • Disk I/O: Monitor disk input/output operations
  • Network Usage: Check network bandwidth usage

Best Practices

Migration Planning

  • Schedule During Low Usage: Run migrations during low-traffic periods
  • Test First: Test migration with small batches first
  • Monitor Resources: Keep an eye on system resources
  • Have Backup: Ensure data backup before migration

Performance Optimization

  • Optimize Batch Size: Find optimal batch size for your system
  • Adjust Delays: Set appropriate delays between batches
  • Monitor CPU: Set realistic CPU thresholds
  • Use Monitoring: Regularly check monitoring data

Security Practices

  • Regular Updates: Keep S3 credentials updated
  • Access Control: Limit admin access to necessary users
  • Audit Logs: Regularly review migration logs
  • Environment Security: Secure environment variable storage

Future Enhancements

Planned Features

  • Incremental Migration: Migrate only changed attachments
  • Parallel Processing: Support for parallel migration streams
  • Advanced Scheduling: Time-based migration scheduling
  • Compression Support: Built-in file compression during migration

Integration Opportunities

  • Cloud Storage: Additional cloud storage providers
  • CDN Integration: Content delivery network support
  • Backup Integration: Automated backup during migration
  • Analytics: Advanced storage analytics and reporting

Support

For issues and questions:

  1. Check this documentation
  2. Review server logs
  3. Use the monitoring tools
  4. Consult the Wekan community
  5. Report issues with detailed information

License

This Enhanced Attachment Migration System is part of Wekan and is licensed under the MIT License.