Data Directory Structure

Legend uses a file-based storage system where all entity data, documentation, and branding assets are stored as plain text files in a structured directory layout. This approach provides transparency, portability, and easy integration with version control systems like Git.

Data Directory Location

The data directory location is controlled by the LEGEND_DATA_DIR environment variable. By default, it is set to /app/data inside the Docker container, which is typically mounted to a host directory for persistence.

Default structure:

• Directorydata/

  • Directorycatalog/ Entity YAML files

    • Directorylandmarks/ Landmark entities

      • …
    • Directorypeople/ Person entities

      • …
    • Directoryteams/ Team entities

      • …
  • Directorydocs/ Documentation files

    • Directorylandmarks/ Documentation for landmarks

      • …
    • Directorypeople/ Documentation for people

      • …
    • Directoryteams/ Documentation for teams

      • …
  • Directorybranding/ Custom branding assets

    • logo.png Custom logo
    • favicon-16.png 16x16 favicon
    • favicon-32.png 32x32 favicon
    • favicon-180.png 180x180 favicon
  • Directorybackups/ Database backups

    • legend-2024-01-15-020000.db
    • legend-2024-01-16-020000.db
    • legend-manual-2024-01-17-143022.db

Catalog Structure

The catalog directory contains YAML files for each entity. Each entity type has its own subdirectory:

Landmarks:

data/catalog/landmarks/
├── billing-api.yaml
├── client-portal.yaml
├── document-hub.yaml
└── identity-provider.yaml

People:

data/catalog/people/
├── alex-johnson.yaml
├── avery-allen.yaml
└── jamie-kim.yaml

Teams:

data/catalog/teams/
├── client-services.yaml
├── operations-and-production.yaml
└── program-delivery.yaml

Entity YAML Structure

Each entity is stored as a YAML file with a standardized structure. Here’s an example of a landmark entity:

id: billing-api
kind: landmark
metadata:
  name: Billing API
  description: Payment processing and invoice management service
  category: service
  icon: credit-card
  tags:
    - production
    - critical
    - payments
  owner:
    type: team
    id: operations-and-production
  links:
    - title: Service Dashboard
      url: https://grafana.example.local/d/billing-api
      icon: chart-line
    - title: Runbook
      url: https://wiki.example.local/runbooks/billing-api
      icon: book
    - title: Repository
      url: https://github.com/example-org/billing-api
      icon: code
  relationships:
    dependsOn:
      - identity-provider
      - document-hub
    dependedOnBy:
      - resource-capacity-planner
    associations: []
  accessGroups: []
  editGroups: []
  pinned: false
  featured: false
  active: true
createdAt: 2024-01-10T14:23:45.123Z
updatedAt: 2024-01-15T09:45:12.456Z

Key fields:

• id: Unique identifier for the entity (used in file names and relationships)
• kind: Entity type (landmark, person, or team)
• metadata: All entity properties
  • name: Display name
  • description: Brief summary
  • category: Landmark category (only for landmarks)
  • icon: FontAwesome icon name or custom URL
  • tags: Array of tag names
  • owner: Owner reference (team or person)
  • links: Array of navigation links
  • relationships: Dependencies and associations
  • accessGroups: Groups that can view this entity
  • editGroups: Groups that can edit this entity
  • pinned: Whether the entity is pinned
  • featured: Whether the entity is featured
  • active: Whether the entity is active
• createdAt: Timestamp when the entity was created
• updatedAt: Timestamp when the entity was last modified

Documentation Files

Documentation files are stored in the docs/ directory, organized by entity type and entity ID:

data/docs/
├── landmarks/
│   ├── billing-api/
│   │   ├── runbook.md
│   │   ├── architecture.md
│   │   └── incident-playbook.md
│   └── client-portal/
│       ├── user-guide.md
│       └── api-docs.md
├── people/
│   └── alex-johnson/
│       └── bio.md
└── teams/
    └── operations-and-production/
        ├── team-charter.md
        └── onboarding.md

Each documentation file is a standard Markdown file with full syntax support:

# Billing API Runbook

## Overview
This runbook covers common operational tasks for the Billing API service.

## Restarting the Service
1. SSH into the production server
2. Run `systemctl restart billing-api`
3. Verify the service is running: `systemctl status billing-api`

## Common Issues

### High Latency
- Check database connection pool
- Review recent deployments
- Monitor CPU and memory usage

### Payment Processing Failures
- Verify payment gateway connectivity
- Check error logs: `journalctl -u billing-api -f`
- Review payment gateway status page

Branding Assets

Custom branding assets are stored in the branding/ directory:

data/branding/
├── logo.png              # Custom logo (uploaded via admin panel)
├── favicon-16.png        # 16x16 favicon
├── favicon-32.png        # 32x32 favicon
└── favicon-180.png       # 180x180 favicon (Apple touch icon)

These files are uploaded through the admin panel and are not typically edited manually. If you need to update branding assets, use the admin interface rather than modifying files directly.

Editing YAML Files Directly

Advanced users can edit YAML files directly using a text editor. This is useful for:

• Bulk updates to multiple entities
• Importing entities from external sources
• Version controlling entity definitions
• Automating entity creation with scripts

To edit YAML files directly:

1. Access the data directory on your host system
2. Navigate to the appropriate entity directory
3. Edit the YAML file using a text editor
4. Save the file
5. Restart the Legend container to reload the data

Direct Editing Risks

Editing YAML files directly bypasses validation and can corrupt your data if done incorrectly. Always back up your data directory before making manual changes. Ensure your YAML syntax is valid and all required fields are present.

Validation requirements:

• Valid YAML syntax
• Required fields present (id, kind, metadata.name)
• Unique entity IDs
• Valid relationship references
• Proper date formats (ISO 8601)

Reloading changes: After editing YAML files, you must restart the Legend container to reload the data:

docker compose restart legend

Or if using Docker Run:

docker restart legend

Version Controlling the Catalog

Because the catalog is stored as plain text YAML files, you can version control it using Git. This provides several benefits:

• Change history: Track who changed what and when
• Collaboration: Multiple people can propose changes through pull requests
• Rollback: Revert to previous versions if needed
• Code review: Review changes before they’re applied
• Automation: Integrate with CI/CD pipelines for automated deployments

Setting up Git version control:

1. Initialize a Git repository in your data directory:

   cd /path/to/legend-data
   git init

2. Create a .gitignore file to exclude files that shouldn’t be version controlled:

   # Exclude backups
   backups/
   
   # Exclude database files
   *.db
   
   # Exclude temporary files
   *.tmp
   *.swp
   
   # Exclude OS files
   .DS_Store
   Thumbs.db

3. Add and commit the catalog:

   git add catalog/ docs/
   git commit -m "Initial catalog commit"

4. Push to a remote repository:

   git remote add origin https://github.com/your-org/legend-catalog.git
   git push -u origin main

Workflow for catalog changes:

1. Create a feature branch:

   git checkout -b add-new-service

2. Make changes to YAML files

3. Commit changes:

   git add catalog/landmarks/new-service.yaml
   git commit -m "Add new service: Payment Gateway"

4. Push and create a pull request:

   git push origin add-new-service

5. After review and merge, pull changes on the production server:

   git pull origin main

6. Restart Legend to reload the catalog:

   docker compose restart legend

Automate Deployments

You can automate catalog deployments by setting up a CI/CD pipeline that pulls changes from your Git repository and restarts the Legend container. This enables GitOps-style management of your Legend instance.

Data Portability

The file-based storage system makes Legend data highly portable:

Migrating to a new server:

1. Copy the entire data directory to the new server
2. Configure the new Legend instance to use the same data directory
3. Start the new instance
4. All data is preserved

Backing up data:

• Copy the data directory to a backup location
• Use your server’s backup tools to include the data directory
• Store backups offsite for disaster recovery

Sharing data between instances:

• Copy specific entity YAML files between instances
• Use Git to synchronize catalogs across environments
• Export and import entities using YAML files

Troubleshooting

Changes not appearing after editing YAML:

• Ensure you saved the file with valid YAML syntax
• Restart the Legend container to reload the data
• Check container logs for parsing errors
• Verify the file is in the correct directory

Entity not loading:

• Check YAML syntax for errors
• Ensure all required fields are present
• Verify the entity ID is unique
• Check that relationship references are valid

Documentation not appearing:

• Ensure the documentation file is in the correct directory
• Verify the file name matches the entity ID
• Check file permissions (must be readable by the container)
• Restart the container to reload documentation

Git conflicts after pulling changes:

• Resolve merge conflicts in YAML files
• Ensure entity IDs remain unique after merge
• Test the catalog locally before deploying
• Restart the container after resolving conflicts

Best Practices

File Organization:

• Use consistent naming conventions for entity IDs (lowercase, hyphenated)
• Keep YAML files well-formatted and readable
• Use comments in YAML files to explain complex configurations
• Organize documentation files logically within entity directories

Version Control:

• Commit changes frequently with descriptive messages
• Use branches for major changes or experiments
• Review pull requests before merging
• Tag releases for important milestones

Backup Strategy:

• Back up the entire data directory regularly
• Store backups offsite for disaster recovery
• Test restore procedures periodically
• Keep multiple backup versions for rollback options

Collaboration:

• Use pull requests for catalog changes
• Document your catalog structure and conventions
• Provide templates for common entity types
• Train team members on YAML syntax and Legend data model

Next Steps

After understanding the data directory structure, proceed to Environment Variables Reference to review all configuration options, or Troubleshooting for help with common issues.