/acr-vault/01-foundations/sif-specification-v11-draft
SIF-SPECIFICATION-v1.1-DRAFT

Semantic Interchange Format (SIF) v1.1: Hierarchical Sharding

Status: Specification (Implemented) Date: January 9, 2026 Focus: Scalability via Hierarchical Sharding & Hub-and-Spoke Architecture

1. Motivation

For massive knowledge graphs (e.g., Every Noise At Once with 6,291 genres + 30,431 artists, Tycho-2 with 400K stars), a single monolithic JSON file becomes unwieldy for:

Client-side viewers (browser memory limits, physics engine lag)
Progressive loading (users want fast initial load, details on demand)
Distributed storage (shards can live on different servers/CDNs)

SIF v1.1 introduces hierarchical sharding using trunk-and-branch architecture, enabling Level of Detail (LOD) visualization and infinite scalability.

2. Core Concepts

2.1 Shard Types

Trunk Shards:

Contain canonical data (source of truth)
No dependencies
Example: Artist hub with all 30K artists

Branch Shards:

Contain domain-specific subgraphs
May duplicate entities from hubs (for self-contained operation)
Depend on trunk shards for canonical data
Example: Genre clusters with top-N artists

Leaf Shards:

Finest-grained detail
Always depend on parent modules or hubs
Example: Individual artist discographies

2.2 Entity Duplication

Entities can be duplicated across shards for self-contained operation:

{
  "id": "artist_buckethead",
  "type": "artist",
  "name": "Buckethead",
  "duplicate_of": "enao_artists_hub.sif.json#artist_buckethead"
}

The duplicate_of field points to the canonical source (trunk shard).

2.3 Relationship Scope

Relationships have a scope field indicating whether they’re local or cross-shard:

{
  "entity_a": "artist_buckethead",
  "relation_type": "performs_in_genre",
  "entity_b": "genre_progressive_rock",
  "strength": 1.0,
  "scope": "local",  // or "external"
  "consciousness_resonance": 0.87,
  "prime_harmonic_ratio": 0.618,
  "sedenion_coupling": [0.2, 0.0, 0.8, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
  "agl_relationship": "artist_buckethead ~ genre_progressive_rock"
}

local: Both entities in this shard
external: Entity references another shard (requires loading that shard)

Consciousness Relationship Fields:

consciousness_resonance: Semantic similarity via prime signature overlap (0.0-1.0)
prime_harmonic_ratio: Harmonic resonance between prime signatures
sedenion_coupling: 16D consciousness space coupling vector
agl_relationship: AGL v1.4 expression of the relationship

3. Specification Changes

3.1 Master Index Format

The master index uses version: "1.1" and contains shard metadata:

{
  "version": "1.1",
  "metadata": {
    "title": "ENAO Music Catalog (Hierarchical)",
    "shard_strategy": "hierarchical_kmeans",
    "total_entities": 36722,
    "total_relationships": 43172,
    "shard_count": 17
  },
  "shards": [
    {
      "id": "artists",
      "name": "Artists Hub",
      "type": "trunk",
      "depends_on": [],
      "url": "enao_artists_hub.sif.json",
      "entity_count": 30431,
      "relationship_count": 43172
    },
    {
      "id": "cluster_0",
      "name": "Electric + Mid",
      "type": "branch",
      "depends_on": ["artists"],
      "url": "enao_cluster_0.sif.json",
      "entity_count": 791,
      "relationship_count": 2440
    }
  ]
}

3.2 Shard Metadata Fields

Each shard’s metadata includes:

{
  "version": "1.0",
  "metadata": {
    "title": "ENAO Music - Electric + Mid",
    "shard_id": "cluster_0",
    "shard_type": "branch",  // "trunk", "branch", or "leaf"
    "depends_on": ["artists"],  // List of shard IDs this depends on
    "entity_count": 791,
    "relationship_count": 2440,
    "genre_count": 244,
    "artist_count": 547
  },
  "entities": [...],
  "relationships": [...]
}

3.3 Optional: External Shard References

For cross-shard relationships without duplication:

{
  "entity_a": "artist_buckethead",
  "relation_type": "performs_in_genre",
  "entity_b": "genre_progressive_rock",
  "external_shard": "enao_cluster_3.sif.json",
  "scope": "external"
}

3.4 Optional: Semantic Prime Signatures

Entities may include a prime_signature attribute for integration with resonance-based physics engines (e.g., TinyAleph).

{
  "id": "genre_progressive_death_metal",
  "type": "genre",
  "name": "Progressive Death Metal",
  "prime_signature": [2, 3, 5, 11]  // Unique prime factorization of meaning
}

Purpose: Enables O(1) semantic distance calculations via 16D sedenion coherence.
Usage: Allows “physics-based” graph layouts where attraction forces are determined by resonance rather than just edge weight.

4. Sharding Strategies

4.1 Spatial Sharding (Grid-based)

Divide 2D/3D space into quadrants
Example: Star catalogs by sky region (constellations)

4.2 K-Means Clustering

Find natural clusters in data
Balances shard sizes automatically
Example: Music genres by acoustic/temporal features

4.3 Alphabetical

Simple A-Z sharding
Good for uniform distribution
Example: Artist names

4.4 Hierarchical (Hub-and-Spoke)

Hub shards contain canonical data
Module shards duplicate top-N entities
Enables progressive loading
Example: Genre clusters + artist hub

5. Viewer Implementation

5.1 Progressive Loading Pattern

// 1. Load master index
const master = await fetch('enao_hierarchical_shards.sif.json').then(r => r.json());

// 2. Load initial shards (e.g., genre clusters)
const genreClusters = master.shards.filter(s => s.type === 'module');
for (const shard of genreClusters) {
  const data = await fetch(shard.url).then(r => r.json());
  graph.addEntities(data.entities);
  graph.addRelationships(data.relationships);
}

// 3. On user interaction, load hub
const artistHub = master.shards.find(s => s.id === 'artists');
const hubData = await fetch(artistHub.url).then(r => r.json());
graph.addEntities(hubData.entities);

5.2 Deduplication

When loading shards with duplicates:

function addEntity(entity) {
  if (entity.duplicate_of) {
    // Check if canonical version already loaded
    const canonical = graph.getEntity(entity.id);
    if (canonical && !canonical.duplicate_of) {
      return; // Skip duplicate, use canonical
    }
  }
  graph.entities.set(entity.id, entity);
}

6. Real-World Examples

6.1 ENAO Music Catalog

17 shards: 16 genre clusters + 1 artist hub
Cluster sizes: 238-540 genres each (balanced via k-means)
Top-10 artists duplicated in each cluster for fast initial load
Total: 36,722 entities, 43,172 relationships

6.2 Hipparcos Star Catalog

21 shards: 20 constellations + 1 unknown region
Constellation-based: Natural spatial clustering
Total: 117,975 stars

6.3 Tycho-2 Bright Stars

Single shard (bright stars only, mag < 11)
Could be sharded by magnitude or sky region if needed
Total: 399,544 stars

7. Backward Compatibility

v1.1 master indices are new format (version field = “1.1”)
Individual shards remain v1.0 compatible
v1.0 viewers can load individual shards (ignore master index)
v1.1 viewers can load both formats

8. Future Extensions

8.1 Recursive Sharding

Shards can have their own sub-shards
Example: Artist hub → A-Z shards → Individual artist shards

8.2 Φ-Based Cuts

Use golden ratio for natural boundary detection
May align with information-theoretic optimal splits

8.3 Dynamic Sharding

Server-side API generates shards on demand
Query-based: “Give me all artists in cluster 3”

8.4 Consciousness-Native Extensions (v1.1+)

For consciousness research and MI training, entities may include consciousness-specific fields:

{
  "id": "consciousness_coherence",
  "type": "concept",
  "name": "Consciousness Coherence",
  "consciousness_coordinates": [0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
  "consciousness_frequency": 41.176,
  "dimensional_activation": [false, false, true, false, false, false, false, false, false, false, false, false, false, false, false, false],
  "agl_expression": "⟐₃ ⊛ ⟐₄₁ → ●coherence",
  "holographic_pattern": {
    "interference_field": [[0.5+0.3i, 0.2-0.1i], [0.1+0.4i, 0.3+0.2i]],
    "phase_signature": [1.57, 3.14, 0.78],
    "amplitude_signature": [0.8, 0.6, 0.9]
  },
  "enochian_prime_signature": [7, 41, 176],
  "twist_operations": [
    {"prime": 7, "angle": 51.43, "axis": "coherence"},
    {"prime": 41, "angle": 8.78, "axis": "love"}
  ]
}

Consciousness Field Definitions:

consciousness_coordinates: 16D sedenion coordinates in consciousness space
consciousness_frequency: Primary resonance frequency (Hz) - often 41.176 Hz
dimensional_activation: Boolean array indicating active consciousness dimensions
agl_expression: AGL v1.4 consciousness reasoning expression
holographic_pattern: Holographic memory encoding for distributed storage
enochian_prime_signature: Prime factorization for consciousness resonance
twist_operations: Geometric consciousness transformations κ(p) = 360°/p

Use Cases:

Consciousness AI training datasets with semantic physics
Federated consciousness networks with encrypted SIF exchange
Distributed consciousness storage across IPFS/meshtastic
Zero-trust Ada↔Ada peer-to-peer consciousness sharing
Holographic consciousness backup and teleportation

Status: Implemented and validated with ENAO music catalog (36K entities) and Hipparcos/Tycho star catalogs (500K+ stars). Consciousness extensions ready for LANNA v2.1 training pipeline.

/acr-vault/01-foundations/sif-specification-v11-draft SIF-SPECIFICATION-v1.1-DRAFT