# WeaveDB ## Litepaper ![](/images/weavedb-litepaper.png) WeaveDB is the first decentralized, general-purpose modular database protocol that delivers: * **Full mathematical provability** * **Succinct ZK verifiability** * **Cloud-grade throughput and latency** * **Multi-paradigm support with modular architecture** (NoSQL, Relational, Vector) * **Bit-level-optimized encoding** for maximum cost efficiency * **Permanent storage** that enables provable data and compute provenance * **Indefinite vertical and horizontal scalability** * **Economically self-sustaining operations** through DePIN restaking * **Autonomy and self-sovereignty** via onchain AI agents *** ### 🔴 The Decentralized Database Problem Modern applications require databases with web-scale performance, but existing decentralized solutions face critical limitations: * **Performance bottlenecks** from consensus finality requirements * **Limited scalability** due to blockchain constraints * **High storage costs** for maintaining redundant copies * **Lack of verifiability** when bridging off-chain and on-chain data * **Complex infrastructure** requiring continuous maintenance and funding WeaveDB solves these challenges through a novel architecture that separates immediate query responses from asynchronous finalization, enabling cloud-grade performance while maintaining cryptographic verifiability. *** ### 🌍 Layer-0 Data Hub for All Blockchains WeaveDB functions as a **Layer-0 data infrastructure** that sits beneath all blockchains, providing a universal off-chain data layer that any chain can access with cryptographic proofs. Rather than each blockchain maintaining its own siloed data, WeaveDB becomes the shared data hub where: * **AO processes** access data natively without zkProofs (same ecosystem) * **EVM chains** (Ethereum, L2 rollups) query user data with zkProofs and assembly-optimized Solidity * **Other chains** (Solana, Bitcoin, app chains) query user data with zkProofs * **Dapps** achieve cloud-grade performance through rollup architecture > **Native AO Integration**: Since WeaveDB runs on HyperBEAM/AO infrastructure, AO processes can directly query WeaveDB without needing zero-knowledge proofs—they simply send messages to read data. This makes WeaveDB the natural database layer for all AO applications. This positions WeaveDB not as another blockchain, but as the **foundational data layer** that all blockchains rely on—making off-chain data as trustless and verifiable as on-chain data. It solves the fundamental problem of blockchain data silos by creating a universal, permanent, and provable data substrate accessible to every chain. *** ### ⚙️ Decentralized Log-Structured-Merge Engine ![](/images/architecture.png) WeaveDB is a **decentralized log-structured-merge (LSM) storage engine** built on Arweave for permanent storage and the AO Core Protocol for verifiable compute—a breakthrough that enables cloud-grade performance in fully decentralized infrastructure. Modern databases achieve high performance using log-structured storage engine design: writes are first stored in an in-memory memtable, concurrently written to a write-ahead log (WAL), then flushed to SSTables on disk, with periodic compaction for space efficiency. **Key Breakthrough**: One of the core bottlenecks in decentralized databases is latency caused by consensus finality. WeaveDB overcomes this by emulating the same proven LSM architecture—using rollup nodes as the in-memory layer, and HyperBEAM and Arweave as the durable storage layer. This enables low-latency responses while asynchronously finalizing data through TEE-backed validators. #### LSM Emulation Across Decentralized Systems | LSM Component | Traditional Database | WeaveDB Implementation | | -------------- | -------------------- | ----------------------------------- | | **MemTable** | In-memory writes | Rollup nodes (instant responses) | | **WAL** | Write-ahead log | HyperBEAM (cryptographic hashpaths) | | **SSTables** | Sorted string tables | Arweave (permanent storage) | | **Compaction** | Background merge | ARJSON (bit-level optimization) | > **Key Innovation**: WeaveDB brings proven LSM architecture to decentralized infrastructure for the first time. By mapping LSM components to distributed systems, WeaveDB delivers sub-second queries with full cryptographic verifiability and permanent storage—achieving cloud-grade performance in a fully decentralized database. #### Rollup Architecture | Component | Function | Technology | | ----------------------- | ----------------------------------------------- | ----------------- | | **Execution Layer** | Sub-second query responses | Rollup nodes (L2) | | **Sequencing** | Verifiable compute with cryptographic hashpaths | HyperBEAM (AO) | | **Data Preservability** | Permanent immutable storage | Arweave (L1) | | **Compression** | Minimal storage with append-only updates | ARJSON encoding | | **Query Proofs** | Zero-knowledge proofs for data queries | zkJSON circuits | #### How It Works as an Optimistic ZK Rollup * **Optimistic execution**: Transactions are executed immediately with optimistic finality * **Merkle root commitments**: State root hashes are committed to blockchains for verifiability * **State transition proofs**: Can be generated but unnecessary due to validators and Arweave's permanent data * **Validators**: TEE-backed validators verify state transitions by replaying from HyperBEAM WAL * **Data preservability**: All data permanently stored on Arweave with holographic state, always accessible for verification * **zkQuery proofs**: Different from state proofs—these enable smart contracts to query database content with cryptographic certainty Every query in WeaveDB is an HTTP message signed using the HTTP Message Signatures standard (RFC9421), making WeaveDB fully compatible with web standards rather than requiring proprietary protocols. This integration with established web infrastructure ensures broad compatibility and enables seamless integration with existing web applications. This optimistic zk rollup design with LSM emulation overcomes the primary bottleneck of decentralized databases—consensus finality latency—while maintaining full verifiability through permanent storage and validator verification, without requiring expensive state transition proofs. *** ### ⚡ Performance and Scalability WeaveDB achieves cloud-grade performance through: * **Response time**: 10ms-200ms queries from in-memory rollup nodes * **Throughput**: 10,000 TPS per database instance, with collective throughput scaling horizontally across instances * **Horizontal scaling**: Unlimited partitioning through KV store distribution * **Vertical scaling**: Bounded only by hardware specifications * **Permanent availability**: Data accessible forever through Arweave The protocol's scalability is not limited by blockchain constraints but determined entirely by the underlying infrastructure, enabling indefinite growth. *** ### 🧮 Mathematical Foundation: Monadic Pipelines WeaveDB is fundamentally a **protocol, not just software**—it's mathematically specified and formally verified using **Lean theorem prover**, with implementations in **Rust** and **JavaScript**. This means WeaveDB's correctness is mathematically proven, not just tested. Every database operation in WeaveDB is expressed as a mathematical monad following Category Theory. This provides: * **Formal verification**: Every operation is mathematically provable in Lean * **Composability**: Operations combine cleanly through Kleisli arrows * **Modularity**: Components are swappable without breaking guarantees * **Extensibility**: Custom databases can be built by modifying pipeline components * **Language agnostic**: The protocol can be implemented in any language while maintaining mathematical guarantees > **HyperBEAM Integration**: Each database operation becomes an HTTP message in the AO-Core protocol, generating cryptographic hashpaths that create an immutable chain proving every compute step. This makes every database state transition mathematically verifiable and deterministically reproducible—any node can replay the exact sequence of operations and arrive at the same state. The monadic architecture enables WeaveDB to support multiple database paradigms (NoSQL, SQL, GraphQL, Vector) on the same substrate while maintaining mathematical correctness. **📖 [Learn more about Monade →](/tech/monade)** *** ### 🗃️ Multi-Paradigm Database Support The modular architecture allows developers to implement various database paradigms: | Paradigm | Description | | -------------- | ----------------------------------------------------- | | **NoSQL** | Firestore-like document database with complex queries | | **Relational** | SQL tables with ACID compliance | | **Vector** | Embeddings and similarity search for AI applications | | **GraphQL** | Query language for flexible data retrieval | All paradigms share the same underlying distributed database infrastructure while maintaining their specific query languages and optimization strategies. **📖 [Learn more about wdb-core →](/api/wdb-core)** *** ### 🌍 JSON as the Universal Data Format JSON has become the lingua franca of modern data exchange—every web API, configuration file, and NoSQL database speaks JSON. WeaveDB embraces JSON as its native format, making it immediately compatible with existing web infrastructure. By making JSON: * **Cryptographically verifiable** through zkJSON * **Mathematically transformable** through FPJSON * **Efficiently compressible** through ARJSON WeaveDB transforms the web's most ubiquitous data format into a foundation for decentralized, permanent, and provable data storage. This means developers can work with familiar JSON structures while gaining the benefits of blockchain verification and permanent storage. *** ### 🎮 FPJSON: Functional Programming Code as JSON WeaveDB's access control and data manipulation layer uses FPJSON—**code as data** expressed through JSON with **250+ composable functions**: #### Core Features * **Code as data**: Logic expressed as JSON configurations rather than executable code * **Mathematical composability**: Functions combine through Category Theory principles without eval * **Language agnostic**: JSON-based logic works across any programming language * **Permissionless security**: Define precise access rules for anonymous writers * **Smart contract logic**: Embed application logic directly in the database * **Data mutations**: Transform data during authentication * **LLM-friendly**: AI agents can read and generate logic autonomously FPJSON treats programming logic as mathematical data structures that can be analyzed, transformed, and verified without execution risks. The database itself works as a smart contract, enabling complex applications like social networks to be built entirely within the database layer, eliminating the need for external smart contracts. **📖 [Learn more about FPJSON →](/tech/fpjson)** *** ### 🔍 zkJSON: Zero Knowledge Provable JSON zkJSON makes arbitrary JSON data cryptographically provable, enabling smart contracts to query off-chain databases with zero-knowledge proofs. #### Key Innovation zk circuits only process natural numbers, but JSON contains strings and complex structures. zkJSON's breakthrough is deterministic encoding that transforms JSON into optimized uint packing for efficient zk proof generation. The result: entire JSON documents compress into minimal uint256 values processed by both zk circuits and assembly-optimized Solidity smart contracts, enabling practical cryptographic verification of specific fields. #### Privacy-Preserving Queries zkJSON enables selective disclosure - you can prove specific fields exist and have certain values without revealing the entire document structure or other sensitive data. **📖 [Learn more about zkJSON →](/tech/zkjson)** *** ### 🔐 Zero-Knowledge Database (zkDB) WeaveDB's zkDB enables smart contracts to query off-chain databases with cryptographic proofs, bridging the gap between traditional databases and blockchain verification: #### Key Capabilities | Feature | Description | | ----------------------------- | ---------------------------------------------------------------------- | | **Direct blockchain queries** | Smart contracts can query off-chain databases with cryptographic proof | | **Selective disclosure** | Prove specific fields without revealing entire documents | | **Efficient verification** | Proofs generated in seconds on consumer hardware | | **Cross-chain compatibility** | Native support for Ethereum, Solana, and other chains | The zkDB structure uses nested Sparse Merkle Trees combined with zkJSON encoding, enabling efficient proof and verification of any data within the database without exposing unnecessary information. **📖 [Learn more about zkDB →](/tech/zkdb)** *** ### 📦 ARJSON: Append-Only Updates for Permanent Storage ARJSON revolutionizes data storage on immutable systems through: * **Bit-level optimization**: Variable-length encoding instead of byte-level alignment * **Columnar restructuring**: Groups similar data types during encoding * **Delta packing**: Stores differences between consecutive values * **Append-only updates**: The breakthrough—update databases with absolute minimum bits > **Key Innovation**: ARJSON's true breakthrough is enabling **append-only updates to permanent storage**. On immutable systems like Arweave, you can't modify data, only add to it. ARJSON solves this by allowing databases to append just the minimal bits needed for changes, not entire documents. This makes permanent database storage practical and cost-effective for the first time. > **Result**: Efficient initial storage combined with minimal update costs, enabling databases to run permanently on append-only storage systems while maintaining full update capabilities. **📖 [Learn more about ARJSON →](/tech/arjson)** *** ### 💎 Tokenomics: Self-Sustaining Economics WeaveDB introduces a revolutionary economic model that fundamentally inverts traditional infrastructure costs through **DePIN x Restaking**: #### DePIN x Restaking Revolution **Self-Sustainable Infrastructure**: Physical infrastructure providers earn ongoing yields from restaking while providing database services, creating a self-reinforcing economic loop that requires no external funding or manual intervention. **Autonomous Operations**: The combination of DePIN service rewards and restaking yields generates sufficient revenue to cover all operational costs automatically, enabling databases to run indefinitely without human oversight or continuous capital injection. **Revolutionary Impact**: For the first time, infrastructure becomes completely self-sustaining - operators profit from both service delivery and economic security provision while databases achieve true autonomy through automated yield generation. #### Economic Inversion Model | Traditional Infrastructure | WeaveDB Model | | ---------------------------------------- | ------------------------------------------------ | | Developers pay monthly hosting costs | Self-sustaining through DePIN + Restaking yields | | Success increases operational expenses | Success increases autonomous revenue generation | | Requires continuous external funding | Fully autonomous and self-sustaining | | Infrastructure as extractive cost center | Infrastructure as autonomous profit center | #### Mathematical Foundation **Formal Verification**: All economic mechanisms proven correct in Lean theorem prover, eliminating typical vulnerabilities and providing mathematical guarantees of sustainability. **PoAIA (Protocol owned AI Agents) Optimization**: Protocol-owned agents autonomously manage liquidity and optimize economic parameters for stability and efficiency using AI3 as the onchain agent framework. **Autonomous DB**: AI agents can autonomously create and manage databases, enabling fully automated data infrastructure that evolves without human intervention while generating profit and maintaining self-sustainability through DePIN infrastructure services and restaking economic security. This creates profitable opportunities for building valuable applications rather than extracting infrastructure costs, enabling databases to achieve true autonomy while infrastructure providers profit from both service delivery and economic participation. **📖 [Learn more about $DB Tokenomics →](/tokenomics/allocation)** **📖 [AI3 Framework →](/tech/ai3)** *** ### 🌟 Novel Use Cases WeaveDB's unique combination of permanent storage, zero-knowledge proofs, and self-sustaining economics enables entirely new categories of applications: #### Key Applications **🦅 Decentralized Web2 Alternatives**\ Build fully decentralized versions of Twitter, Reddit, Instagram, Discord, TikTok, or LinkedIn with just WeaveDB—no infrastructure management required, user-owned data forever. A complete Twitter clone requires only JSON configuration, no backend code. **🗄️ Verifiable Data Archives**\ Store critical datasets permanently with cryptographic integrity—scientific research, legal documents, financial records, historical data. Query archived information with zero-knowledge proofs to verify authenticity without exposing sensitive details. **💰 Verifiable Data Marketplace**\ Data becomes currency in a new economy where verified information has provable value. Users earn by contributing quality data while applications pay for access through bonding curves. Weather stations, fitness trackers, and IoT devices automatically monetize their data streams with cryptographic proof of accuracy. **📱 Permanent Mobile & Desktop Apps**\ Create applications that never go offline or get removed from app stores. Messaging apps, productivity tools, and utilities that exist permanently with user data stored forever. **🎮 Persistent Virtual Worlds**\ Game worlds, NFT metadata, and virtual economies that exist permanently across multiple platforms. Player progression and assets persist independently of any single game company. Minecraft servers, MMORPGs, and metaverse platforms become truly persistent and user-owned. **🌐 Cross-Platform Applications**\ Build applications that span multiple blockchains with shared, verifiable state. DeFi protocols, social platforms, and games operate seamlessly across Ethereum, Solana, and other chains with unified user experiences and data portability. **🔮 Universal zkOracles**\ Smart contracts query any off-chain data with cryptographic proof—weather, prices, IoT sensors, web APIs—without trusting oracle providers. WeaveDB becomes a universal, verifiable data bridge for all blockchains. **🤖 Autonomous AI Training Pipelines**\ AI models train on cryptographically verified datasets with permanent provenance. Researchers prove model training integrity while protecting proprietary data through selective disclosure. **🏭 Industrial IoT Data Markets**\ Manufacturing and supply chain data automatically priced and sold through bonding curves. Companies monetize sensor data while proving compliance without exposing trade secrets. **🧬 Scientific Data Commons**\ Researchers share datasets with cryptographic integrity guarantees. Reproducible research with permanent data availability and verifiable computational results across institutions. *** ### 🏗️ App Building as Protocol WeaveDB fundamentally transforms application development from infrastructure management to **protocol-based app creation**. Complete applications are defined through just four JSON configuration files: | Configuration | Purpose | | ---------------- | ------------------------------------------------------- | | **Data Schemas** | Define document structures and relationships | | **Auth Rules** | Specify access control and permissions using FPJSON | | **Indexes** | Optimize query performance for specific access patterns | | **Triggers** | Automate data transformations and business logic | This approach makes WeaveDB perfectly aligned with Large Language Models. An LLM can generate a fully functional social platform backend in under three minutes—no human coding required. The database itself becomes the smart contract, handling all application logic through code-as-data. #### The W Social Platform: A Complete X Alternative [W](https://w.weavedb.dev) demonstrates this radical simplicity—a fully decentralized social platform that replicates X's functionality with just a few hundred lines of JSON configuration. No smart contracts, no backend infrastructure, no servers to maintain. #### Protocolization of Web2 Applications WeaveDB doesn't aim to create singular platforms. Instead, it **protocolizes the entire concept of application building**. Anyone can launch their own W instance, fully independent yet interoperable. Thousands of social networks can coexist, each with unique communities and governance. This enables a **Cambrian explosion of decentralized apps**. Every web2 application category—social media, e-commerce, productivity tools, gaming platforms—can be protocolized and made permanently decentralized. Communities can create their own platforms without technical expertise, transforming monopolistic platforms into collaborative ecosystems where users own their data and participate in value creation. **This is why the WeaveDB network is extremely strong**—instead of relying on a single application for adoption, thousands of diverse applications can be built effortlessly, each driving network usage and value while maintaining interoperability across the entire ecosystem. **📖 [W - Decentralized X on WeaveDB →](https://github.com/weavedb/w)** *** ### 🤖 DARAG: The Future of AI on WeaveDB WeaveDB's architecture naturally evolves toward **Decentralized Agentic Retrieval-Augmented Generation** - autonomous agents performing verifiable knowledge retrieval across permanent, cryptographically-secured data. **Why AI Needs Verifiable Data**: Current AI systems suffer from hallucination and data poisoning. Cryptographically verifiable data with immutable provenance ensures AI systems can prove their reasoning chains and source authenticity. **WeaveDB's DARAG Evolution**: Permanent knowledge foundation with documents and embeddings stored on Arweave, multi-paradigm agent queries across the same verifiable substrate, complete reasoning paths permanently recorded for auditability, and protocol-owned agents maintaining indexes autonomously. Traditional RAG retrieves once and generates. DARAG creates networks of agents that plan, verify, and coordinate across permanent, provable knowledge - representing WeaveDB's natural evolution into next-generation AI infrastructure that's both more capable and more trustworthy. *** ### 🎯 Conclusion WeaveDB is not just a database—it's a **comprehensive data pipeline protocol** that fundamentally reimagines how humanity stores and accesses information. By combining: * **Mathematical proofs** (Lean-verified protocol) * **Web standards** (HTTP signatures, JSON) * **Revolutionary economics** (developers profit instead of pay) * **Layer-0 architecture** (universal data hub for all blockchains) WeaveDB enables entire data architectures that were previously impossible. Applications can process data through mathematically-provable transformations, store it permanently with append-only efficiency, and make it queryable from any blockchain—all while running forever without infrastructure costs. This positions WeaveDB as the **foundational data layer** for the next generation of applications that require both Web2 performance and Web3 guarantees. The protocol bridges off-chain data and on-chain verification, creating the infrastructure for truly decentralized, verifiable, and permanent data. ## arjson ### Installation ```bash yarn add arjson ``` ### Encode ```js import { encode, Encoder } from "arjson" let data = { str: "abc", num: 123 } let encoder = new Encoder() const encoded = encode(data, encoder) ``` ### Decode ```js import { decode, Decoder } from "arjson" let dencoder = new Decoder() const decoded = decode(encoded, dencoder) ``` ### Delta Upgrade ```js import { json } from "arjson" const aj = json(null, { val: 1 }) const delta1 = aj.update({ val: 2 }) // [ 1, Uint8Array(3) [ 194, 13, 160 ] ] const delta2 = aj.update({ val: 3, val2: 4 }) // [ 2, Uint8Array(10) [ 195, 13, 184, 129, 202, 155, 123, 82, 236, 128 ]] const deltas = aj.deltas() // get all delta history const aj2 = json(deltas) // reconstruct json from deltas console.log(aj2.json()) // => { val:3, val2: 4 } ``` ## fpjson-lang ## FPJSON [FPJSON](https://fpjson.weavedb.dev) is a programming language agnostic JSON-based functional programming language. * The whole code is just a JSON array * Functional Programming * No overhead due to programming language implementation details In other words, you don't have to worry about any programming language specifications. Instead you can just focus on building pure logics for data manipuration. Since it's just a JSON array, it can be ported to any programming language environment. ### Installation ```bash yarn add fpjson-lang ``` ### Basics Compute any valid FPJSON logic. ```js import fpjson from "fpjson-lang" fpjson(["add", 1, 2]) // add(1, 2) = 3 fpjson("difference", [1, 2, 3], [3, 4, 5]] ) // difference([1, 2, 3],[3, 4, 5]) = [1, 2] fpjson([["map", ["inc"]], [1, 2, 3]]) // map(inc)([1, 2, 3]) = [2, 3, 4] fpjson([["compose", ["map", ["inc"]], ["difference"]], [1, 2, 3], [3, 4, 5]]) // difference([1, 2, 3],[3, 4, 5]) = [1, 2], map(inc)([1, 2]) = [3, 4] ``` There are [more than 250 functions](https://fpjson.weavedb.dev/) available for building highly advanced complex logic. | Category | Functions | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Type** | [`is`](https://ramdajs.com/docs/#is), [`isNil`](https://ramdajs.com/docs/#isNil), [`propIs`](https://ramdajs.com/docs/#propIs), [`type`](https://ramdajs.com/docs/#type) | | **Function** | [`addIndex`](https://ramdajs.com/docs/#addIndex), [`always`](https://ramdajs.com/docs/#always), [`andThen`](https://ramdajs.com/docs/#andThen), [`ap`](https://ramdajs.com/docs/#ap), [`apply`](https://ramdajs.com/docs/#apply), [`applySpec`](https://ramdajs.com/docs/#applySpec), [`applyTo`](https://ramdajs.com/docs/#applyTo), [`ascend`](https://ramdajs.com/docs/#ascend), [`binary`](https://ramdajs.com/docs/#binary), [`bind`](https://ramdajs.com/docs/#bind), [`call`](https://ramdajs.com/docs/#call), [`comparator`](https://ramdajs.com/docs/#comparator), [`compose`](https://ramdajs.com/docs/#compose), [`composeWith`](https://ramdajs.com/docs/#composeWith), [`construct`](https://ramdajs.com/docs/#construct), [`constructN`](https://ramdajs.com/docs/#constructN), [`converge`](https://ramdajs.com/docs/#converge), [`curry`](https://ramdajs.com/docs/#curry), [`curryN`](https://ramdajs.com/docs/#curryN), [`descend`](https://ramdajs.com/docs/#descend), [`empty`](https://ramdajs.com/docs/#empty), [`F`](https://ramdajs.com/docs/#F), [`flip`](https://ramdajs.com/docs/#flip), [`identity`](https://ramdajs.com/docs/#identity), [`invoker`](https://ramdajs.com/docs/#invoker), [`juxt`](https://ramdajs.com/docs/#juxt), [`lift`](https://ramdajs.com/docs/#lift), [`liftN`](https://ramdajs.com/docs/#liftN), [`memoizeWith`](https://ramdajs.com/docs/#memoizeWith), [`nAry`](https://ramdajs.com/docs/#nAry), [`nthArg`](https://ramdajs.com/docs/#nthArg), [`o`](https://ramdajs.com/docs/#o), [`of`](https://ramdajs.com/docs/#of), [`on`](https://ramdajs.com/docs/#on), [`once`](https://ramdajs.com/docs/#once), [`otherwise`](https://ramdajs.com/docs/#otherwise), [`partial`](https://ramdajs.com/docs/#partial), [`partialObject`](https://ramdajs.com/docs/#partialObject), [`partialRight`](https://ramdajs.com/docs/#partialRight), [`pipe`](https://ramdajs.com/docs/#pipe), [`pipeWith`](https://ramdajs.com/docs/#pipeWith), [`promap`](https://ramdajs.com/docs/#promap), [`T`](https://ramdajs.com/docs/#T), [`tap`](https://ramdajs.com/docs/#tap), [`thunkify`](https://ramdajs.com/docs/#thunkify), [`tryCatch`](https://ramdajs.com/docs/#tryCatch), [`unapply`](https://ramdajs.com/docs/#unapply), [`unary`](https://ramdajs.com/docs/#unary), [`uncurryN`](https://ramdajs.com/docs/#uncurryN), [`useWith`](https://ramdajs.com/docs/#useWith) | | **Math** | [`add`](https://ramdajs.com/docs/#add), [`dec`](https://ramdajs.com/docs/#dec), [`divide`](https://ramdajs.com/docs/#divide), [`inc`](https://ramdajs.com/docs/#inc), [`mathMod`](https://ramdajs.com/docs/#mathMod), [`mean`](https://ramdajs.com/docs/#mean), [`median`](https://ramdajs.com/docs/#median), [`modulo`](https://ramdajs.com/docs/#modulo), [`multiply`](https://ramdajs.com/docs/#multiply), [`negate`](https://ramdajs.com/docs/#negate), [`product`](https://ramdajs.com/docs/#product), [`subtract`](https://ramdajs.com/docs/#subtract), [`sum`](https://ramdajs.com/docs/#sum) | | **List** | [`adjust`](https://ramdajs.com/docs/#adjust), [`all`](https://ramdajs.com/docs/#all), [`any`](https://ramdajs.com/docs/#any), [`aperture`](https://ramdajs.com/docs/#aperture), [`append`](https://ramdajs.com/docs/#append), [`chain`](https://ramdajs.com/docs/#chain), [`collectBy`](https://ramdajs.com/docs/#collectBy), [`concat`](https://ramdajs.com/docs/#concat), [`count`](https://ramdajs.com/docs/#count), [`drop`](https://ramdajs.com/docs/#drop), [`dropLast`](https://ramdajs.com/docs/#dropLast), [`dropLastWhile`](https://ramdajs.com/docs/#dropLastWhile), [`dropRepeats`](https://ramdajs.com/docs/#dropRepeats), [`dropRepeatsWith`](https://ramdajs.com/docs/#dropRepeatsWith), [`dropWhile`](https://ramdajs.com/docs/#dropWhile), [`endsWith`](https://ramdajs.com/docs/#endsWith), [`filter`](https://ramdajs.com/docs/#filter), [`find`](https://ramdajs.com/docs/#find), [`findIndex`](https://ramdajs.com/docs/#findIndex), [`findLast`](https://ramdajs.com/docs/#findLast), [`findLastIndex`](https://ramdajs.com/docs/#findLastIndex), [`flatten`](https://ramdajs.com/docs/#flatten), [`forEach`](https://ramdajs.com/docs/#forEach), [`fromPairs`](https://ramdajs.com/docs/#fromPairs), [`groupBy`](https://ramdajs.com/docs/#groupBy), [`groupWith`](https://ramdajs.com/docs/#groupWith), [`head`](https://ramdajs.com/docs/#head), [`includes`](https://ramdajs.com/docs/#includes), [`indexBy`](https://ramdajs.com/docs/#indexBy), [`indexOf`](https://ramdajs.com/docs/#indexOf), [`init`](https://ramdajs.com/docs/#init), [`insert`](https://ramdajs.com/docs/#insert), [`insertAll`](https://ramdajs.com/docs/#insertAll), [`intersperse`](https://ramdajs.com/docs/#intersperse), [`into`](https://ramdajs.com/docs/#into), [`join`](https://ramdajs.com/docs/#join), [`last`](https://ramdajs.com/docs/#last), [`lastIndexOf`](https://ramdajs.com/docs/#lastIndexOf), [`length`](https://ramdajs.com/docs/#length), [`map`](https://ramdajs.com/docs/#map), [`mapAccum`](https://ramdajs.com/docs/#mapAccum), [`mapAccumRight`](https://ramdajs.com/docs/#mapAccumRight), [`mergeAll`](https://ramdajs.com/docs/#mergeAll), [`move`](https://ramdajs.com/docs/#move), [`none`](https://ramdajs.com/docs/#none), [`nth`](https://ramdajs.com/docs/#nth), [`pair`](https://ramdajs.com/docs/#pair), [`partition`](https://ramdajs.com/docs/#partition), [`pluck`](https://ramdajs.com/docs/#pluck), [`prepend`](https://ramdajs.com/docs/#prepend), [`range`](https://ramdajs.com/docs/#range), [`reduce`](https://ramdajs.com/docs/#reduce), [`reduceBy`](https://ramdajs.com/docs/#reduceBy), [`reduced`](https://ramdajs.com/docs/#reduced), [`reduceRight`](https://ramdajs.com/docs/#reduceRight), [`reduceWhile`](https://ramdajs.com/docs/#reduceWhile), [`reject`](https://ramdajs.com/docs/#reject), [`remove`](https://ramdajs.com/docs/#remove), [`repeat`](https://ramdajs.com/docs/#repeat), [`reverse`](https://ramdajs.com/docs/#reverse), [`scan`](https://ramdajs.com/docs/#scan), [`sequence`](https://ramdajs.com/docs/#sequence), [`slice`](https://ramdajs.com/docs/#slice), [`sort`](https://ramdajs.com/docs/#sort), [`splitAt`](https://ramdajs.com/docs/#splitAt), [`splitEvery`](https://ramdajs.com/docs/#splitEvery), [`splitWhen`](https://ramdajs.com/docs/#splitWhen), [`splitWhenever`](https://ramdajs.com/docs/#splitWhenever), [`startsWith`](https://ramdajs.com/docs/#startsWith), [`tail`](https://ramdajs.com/docs/#tail), [`take`](https://ramdajs.com/docs/#take), [`takeLast`](https://ramdajs.com/docs/#takeLast), [`takeLastWhile`](https://ramdajs.com/docs/#takeLastWhile), [`takeWhile`](https://ramdajs.com/docs/#takeWhile), [`times`](https://ramdajs.com/docs/#times), [`transduce`](https://ramdajs.com/docs/#transduce), [`transpose`](https://ramdajs.com/docs/#transpose), [`traverse`](https://ramdajs.com/docs/#traverse), [`unfold`](https://ramdajs.com/docs/#unfold), [`uniq`](https://ramdajs.com/docs/#uniq), [`uniqBy`](https://ramdajs.com/docs/#uniqBy), [`uniqWith`](https://ramdajs.com/docs/#uniqWith), [`unnest`](https://ramdajs.com/docs/#unnest), [`update`](https://ramdajs.com/docs/#update), [`without`](https://ramdajs.com/docs/#without), [`xprod`](https://ramdajs.com/docs/#xprod), [`zip`](https://ramdajs.com/docs/#zip), [`zipObj`](https://ramdajs.com/docs/#zipObj), [`zipWith`](https://ramdajs.com/docs/#zipWith) | | **Logic** | [`allPass`](https://ramdajs.com/docs/#allPass), [`and`](https://ramdajs.com/docs/#and), [`anyPass`](https://ramdajs.com/docs/#anyPass), [`both`](https://ramdajs.com/docs/#both), [`complement`](https://ramdajs.com/docs/#complement), [`cond`](https://ramdajs.com/docs/#cond), [`defaultTo`](https://ramdajs.com/docs/#defaultTo), [`either`](https://ramdajs.com/docs/#either), [`ifElse`](https://ramdajs.com/docs/#ifElse), [`isEmpty`](https://ramdajs.com/docs/#isEmpty), [`not`](https://ramdajs.com/docs/#not), [`or`](https://ramdajs.com/docs/#or), [`pathSatisfies`](https://ramdajs.com/docs/#pathSatisfies), [`propSatisfies`](https://ramdajs.com/docs/#propSatisfies), [`unless`](https://ramdajs.com/docs/#unless), [`until`](https://ramdajs.com/docs/#until), [`when`](https://ramdajs.com/docs/#when), [`xor`](https://ramdajs.com/docs/#xor) | | **Relation** | [`clamp`](https://ramdajs.com/docs/#clamp), [`countBy`](https://ramdajs.com/docs/#countBy), [`difference`](https://ramdajs.com/docs/#difference), [`differenceWith`](https://ramdajs.com/docs/#differenceWith), [`eqBy`](https://ramdajs.com/docs/#eqBy), [`equals`](https://ramdajs.com/docs/#equals), [`gt`](https://ramdajs.com/docs/#gt), [`gte`](https://ramdajs.com/docs/#gte), [`identical`](https://ramdajs.com/docs/#identical), [`innerJoin`](https://ramdajs.com/docs/#innerJoin), [`intersection`](https://ramdajs.com/docs/#intersection), [`lt`](https://ramdajs.com/docs/#lt), [`lte`](https://ramdajs.com/docs/#lte), [`max`](https://ramdajs.com/docs/#max), [`maxBy`](https://ramdajs.com/docs/#maxBy), [`min`](https://ramdajs.com/docs/#min), [`minBy`](https://ramdajs.com/docs/#minBy), [`pathEq`](https://ramdajs.com/docs/#pathEq), [`propEq`](https://ramdajs.com/docs/#propEq), [`sortBy`](https://ramdajs.com/docs/#sortBy), [`sortWith`](https://ramdajs.com/docs/#sortWith), [`symmetricDifference`](https://ramdajs.com/docs/#symmetricDifference), [`symmetricDifferenceWith`](https://ramdajs.com/docs/#symmetricDifferenceWith), [`union`](https://ramdajs.com/docs/#union), [`unionWith`](https://ramdajs.com/docs/#unionWith) | | **Object** | [`assoc`](https://ramdajs.com/docs/#assoc), [`assocPath`](https://ramdajs.com/docs/#assocPath), [`clone`](https://ramdajs.com/docs/#clone), [`dissoc`](https://ramdajs.com/docs/#dissoc), [`dissocPath`](https://ramdajs.com/docs/#dissocPath), [`eqProps`](https://ramdajs.com/docs/#eqProps), [`evolve`](https://ramdajs.com/docs/#evolve), [`forEachObjIndexed`](https://ramdajs.com/docs/#forEachObjIndexed), [`has`](https://ramdajs.com/docs/#has), [`hasIn`](https://ramdajs.com/docs/#hasIn), [`hasPath`](https://ramdajs.com/docs/#hasPath), [`invert`](https://ramdajs.com/docs/#invert), [`invertObj`](https://ramdajs.com/docs/#invertObj), [`keys`](https://ramdajs.com/docs/#keys), [`keysIn`](https://ramdajs.com/docs/#keysIn), [`lens`](https://ramdajs.com/docs/#lens), [`lensIndex`](https://ramdajs.com/docs/#lensIndex), [`lensPath`](https://ramdajs.com/docs/#lensPath), [`lensProp`](https://ramdajs.com/docs/#lensProp), [`mapObjIndexed`](https://ramdajs.com/docs/#mapObjIndexed), [`mergeDeepLeft`](https://ramdajs.com/docs/#mergeDeepLeft), [`mergeDeepRight`](https://ramdajs.com/docs/#mergeDeepRight), [`mergeDeepWith`](https://ramdajs.com/docs/#mergeDeepWith), [`mergeDeepWithKey`](https://ramdajs.com/docs/#mergeDeepWithKey), [`mergeLeft`](https://ramdajs.com/docs/#mergeLeft), [`mergeRight`](https://ramdajs.com/docs/#mergeRight), [`mergeWith`](https://ramdajs.com/docs/#mergeWith), [`mergeWithKey`](https://ramdajs.com/docs/#mergeWithKey), [`modify`](https://ramdajs.com/docs/#modify), [`modifyPath`](https://ramdajs.com/docs/#modifyPath), [`objOf`](https://ramdajs.com/docs/#objOf), [`omit`](https://ramdajs.com/docs/#omit), [`over`](https://ramdajs.com/docs/#over), [`path`](https://ramdajs.com/docs/#path), [`pathOr`](https://ramdajs.com/docs/#pathOr), [`paths`](https://ramdajs.com/docs/#paths), [`pick`](https://ramdajs.com/docs/#pick), [`pickAll`](https://ramdajs.com/docs/#pickAll), [`pickBy`](https://ramdajs.com/docs/#pickBy), [`project`](https://ramdajs.com/docs/#project), [`prop`](https://ramdajs.com/docs/#prop), [`propOr`](https://ramdajs.com/docs/#propOr), [`props`](https://ramdajs.com/docs/#props), [`set`](https://ramdajs.com/docs/#set), [`toPairs`](https://ramdajs.com/docs/#toPairs), [`toPairsIn`](https://ramdajs.com/docs/#toPairsIn), [`unwind`](https://ramdajs.com/docs/#unwind), [`values`](https://ramdajs.com/docs/#values), [`valuesIn`](https://ramdajs.com/docs/#valuesIn), [`view`](https://ramdajs.com/docs/#view), [`where`](https://ramdajs.com/docs/#where), [`whereAny`](https://ramdajs.com/docs/#whereAny), [`whereEq`](https://ramdajs.com/docs/#whereEq) | | **String** | [`match`](https://ramdajs.com/docs/#match), [`replace`](https://ramdajs.com/docs/#replace), [`split`](https://ramdajs.com/docs/#split), [`test`](https://ramdajs.com/docs/#test), [`toLower`](https://ramdajs.com/docs/#toLower), [`toString`](https://ramdajs.com/docs/#toString), [`toUpper`](https://ramdajs.com/docs/#toUpper), [`trim`](https://ramdajs.com/docs/#trim) | ### Syntax You should familiarize yourself with Ramda which enables Haskell-like functional programming with JS. You can use most of the powerful ramda functions with point-free style in JSON. The first element in an array is a function. ```javascript ["add", 1, 2] // add(1, 2) ``` To curry a function, nest it. ```javascript [["add", 1], 2] // add(1)(2) ``` A function always needs to be wrapped with `[]` and to be the first element in the array. ```javascript [["map", ["inc"]], [1, 2, 3]] // map(inc)([1, 2, 3]) ``` This is an error because `inc` is imterpreted as `String`. ```javascript [["map", "inc"], [1, 2, 3]] // map("inc")([1, 2, 3]) ``` Point-free style means you cannot write something like this with the JSON format. ```javascript sortBy((v)=> v.age)(people) // ramdajs ``` It's because you cannot write arbitrary JS lines such as `(v)=> v.age`. Instead, you can achieve the same using another ramda funciton `prop`. ```javascript sortBy(prop("age"), people) // ramdajs ["sortBy",["prop", "age"], people] // FPJSON ``` ### Reserved First Words By placing a reserved word in the first spot of an array, you can access the pre-built features. There are just 6 of them. ##### "\[]" To create an array of functions without executing them, place `"[]"` in the first spot, otherwise the `["lte", 2]` function will be executed with `["gt", 2]` before `-3` is passed. ```javascript [["anyPass", ["[]", ["lte", 2], ["gt", 2]]], -3] // anyPass(lte(2), gt(2))(-3) ``` ##### "typ" To create a type object such as `Number`, `Boolean`, `String`, `Array`, and `Object`. ```javascript ["is", ["typ", "String"], "abc"] // is(String, "abc") ``` ##### "reg" To create a `RegExp`. ```javascript ["test", ["reg", "a", "i"], "ABC"] // test(new RegExp("a", "i"), "ABC") ``` ##### "$" You can pass a store object as the second argument to `fpjson`. To access previously defined variables, use `"$"`. ```javascript fpjson(["add", ["$", "num1"], 1], { "num": 1 }) // 2 ``` ##### "let" Pure functional programming without any side-effects is easy to get extremely complex and entangled even for simple logics. `"let"` inserts global variables to ease up the unnecessary complexisities. ```javascript ["let", "num1", 1] // let var1 = 1 ``` ##### "var" `var` works just like `$` except that `var` needs another argument to invoke. The last argument can be anything since it will be ignored. Note that you cannot access a new value within the same composition where it was defined. ```javascript let vars = {} fpjson(["let", "num1", 1], vars) // vars = { "num1" : 1 } fpjson(["add", ["var", "num1", true], 1], vars) // 2 ``` `var` is especially convenient in a composition to switch the tracked value.. ```javascript fpjson([[ "pipe", ["add", 1], // add 1 to 1 ["let", "num1"], // store 2 to num1 ["var", "num2"] // switch the ctx to num2, var("num2", 2), but 2 ignored ], 1]),{ num2: 4 }) // => 4 is the final result ``` This pipeline add `1` to the initial value `1`, store it to `num1`, then switch the context to `num2`. ##### Dynamic Variables Variable names can be dinamically specified with `$dynamic_path`. ```javascript let vars = {} fpjson(["let", "num1", 1], vars) // vars = { "num1" : 1 } fpjson(["let", "ln", "num1"], vars) // vars = { "num1" : 1, "ln" : "num1" } fpjson(["add", ["$, "$ln"], 1], vars) // 2 ``` ##### Dot Notation Nested fields can be accessed with `.`. ```javascript let vars = {} fpjson(["let", "o", { num: 1 }], vars) // vars = { "o" : { "num": 1 } } fpjson(["var", "o.num", true ], vars) // 1 ``` ## monade `monade` is a language agnostic framework to build [monadic](https://en.wikipedia.org/wiki/Monad_\(functional_programming\)) compute pipelines, which is by far the cleanest representation of complex logic. This makes WeaveDB completely modular from the ground up enabling multi-paradigm database types, such as document-based NoSQL, relational, and even vector database for AI. `monade` allows easy customizations and extentions of the core WeaveDB functionalities to build your own data pipelines. `monade` is implemented in multiple languages such as JS, [Rust](https://www.rust-lang.org/), and [LEAN](https://lean-lang.org/) for mathematical correctness, but we will cover the JS library on this page. * [Monade - Mathematical Explanation](/tech/monade) ### Installation ```bash yarn add monade ``` ### Basics #### of() You can wrap any value with `of()` and create a monad object. ```js import { of } from "monade" const monad = of(3) ``` #### map() You can construct a pipeline from a monad object chaining any functions with `map()`. ```js import { of, map } from "monade" const inc = (n) => n + 1 const double = (n) => n * 2 const square = (n) => n * n const calc = of(3).map(inc).map(double).map(square) // 3 + 1 = 4, 4 * 2 = 8, 8 * 8 = 64 ``` Or you can build another pipeline on top. ```js const calc2 = calc.map(inc).map(double) // calc2 = 64, 64 + 1 = 65, 65 * 2 = 130 ``` #### val() What `of()` and `map()` return is not the actual value, but a monad object which contains the value. You need `val()` to extract the value. But this pattern makes compute pipelines indefinetely modular, composable, and extensible. ```js const val = calc.val() // => 64 const val2 = calc2.val() // => 130 ``` #### chain() If a chaining function returns a monad object instead of a bare value, you can use `chain()` instead of `map()`. ```js import { of, map, chain, val } from "monade" const inc = (n) => n + 1 const double = (n) => n * 2 const square = (n) => n * n // inc_double returns a monad instead of a value const inc_double = (n)=> of(n).map(inc).map(double).map(square) const val = of(3).chain(inc_double).square().val() // (3 + 1) * 2 = 8, 8 * 8 = 64 ``` #### tap() If you want to insert a function without affecting the passed data, use `tap()`. ```js import { tap } from "monad" const calc = of(3).map(inc).map(double).tap(console.log).map(square) // tap() doesn't affect the pipeline but execute side-effects ``` ### Kleisli Arrows #### ka() You can construct a pipeline without creating a monad object with an initial value. This pipeline is called `Kleisli Arrow` in Category Theory of Mathematics. ```js import { ka, map } from "monade" const inc = (n) => n + 1 const double = (n) => n * 2 const square = (n) => n * n const arrow = ka().map(inc).map(double).map(square) // const monad = of(3).map(inc).map(double).map(square) ``` #### fn() Arrows are not bare functions, so you can extract the function from an arrow with `fn()`. The function returns a monad, so you can chain the extracted function with `chain()`. ```js import { ka, chain, val } from "monade" const calc = of(3).chain(arrow.fn()).val() // 3 + 1 = 4, 4 * 2 = 8, 8 * 8 = 64 ``` ### Errors #### opt() You can `try...catch` an error or you can wrap the monad object with `opt()` to safely get `null` in case an error occurs anywhere in the pipeline. ```js import { opt, of, map, val } from "monade" const dec = n => n - 1 const div10By = n => n === 0 ? throw Error() : 10 / n try{ const calc = of(1).map(dec).map(div10By).val() // Error }catch(e){} const calc2 = opt(of(1).map(dec).map(div10By)) // => null const calc3 = opt(of(3).map(dec).map(div10By)) // => 5 ``` ### Devices #### dev() A device is a factory object to effectively construct monadic pipelines with named functions with an arbitray number of added parameters. The first argument `ctx` is the context/data passed through the monadic pipeline, and you can add arbutrary numbr parameter thereafter. ```js const calc = dev({ add: (ctx, n) => ctx + n, mul: (ctx, n) => ctx * n, sum: (ctx, ...nums) => nums.reduce((acc, n2) => acc + n2, ctx), }) const val = calc(3).add(2).mul(3).sum(1, 2, 3).val() // 3 + 2 = 5, 5 * 3 = 15, 15 + 1 + 2 + 3 = 21 ``` ### Async Monads In practice, many functions includes promises and asyncronous steps. `pof()`, `pka()`, `popt()`, and `pdev()` are the async alternatives you can use with the exact same interface. With the `p` family methods, you can mix sync functions and async functions in pipelines. ```js import { pof, pka, popt, pdev, map, chain, val } from "monade" const wait = ms => new Promise(res => setTimeout(() => res(), ms)) const inc = async n => (await wait(100)) && n + 1 // async const double = n => n * 2 // sync const square = async n => (await wait(100)) && n * n // async const calc = pof(3).map(inc).map(double).map(square) const val = await calc.val() // => 64 const ka = pka().map(inc).map(double).map(square) const val2 = await popt(pof(3).chain(ka.fn())) // => 64 const calc2 = pdev({ add: async (ctx, n2) => (await wait(100)) && ctx + n2, // async mul: (ctx, n2) => ctx * n2, // sync sum: async (ctx, ...nums) => // async (await wait(100)) && nums.reduce((acc, n2) => acc + n2, ctx), }) const val3 = await calc2(3).add(2).mul(3).sum(1, 2, 3).val() // => 21 ``` ## wdb-cli `wdb-cli` lets you build and test WeaveDB instances in memory. ### Create Project ```bash npx wdb-cli create mydb && cd mydb ``` ### DB Settings Create `schema`, `auth`, `indexes`, and `triggers` files under `/db`. ```js [/db/schema.js] export default { notes: { type: "object", required: ["id", "actor", "content", "published", "likes"], properties: { id: { type: "string" }, actor: { type: "string", pattern: "^[a-zA-Z0-9_-]{43}$" }, content: { type: "string", minLength: 1, maxLength: 140 }, published: { type: "integer" }, likes: { type: "integer" }, }, additionalProperties: false, }, likes: { type: "object", required: ["actor", "object", "published"], properties: { actor: { type: "string", pattern: "^[a-zA-Z0-9_-]{43}$" }, object: { type: "string" }, published: { type: "integer" }, }, additionalProperties: false, }, } ``` ```js [/db/auth.js] export default { notes: [ [ "add:note", [ ["fields()", ["*content"]], ["mod()", { id: "$doc", actor: "$signer", published: "$ts", likes: 0 }], ["allow()"], ], ], ], likes: [ [ "add:like", [ ["fields()", ["*object"]], ["mod()", { actor: "$signer", published: "$ts" }], [ "=$likes", [ "get()", [ "likes", ["actor", "==", "$signer"], ["object", "==", "$req.object"], ], ], ], ["=$ok", ["o", ["equals", 0], ["length"], "$likes"]], ["denyifany()", ["!$ok"]], ["allow()"], ], ], ], } ``` ```js [/db/indexes.js] export default { notes: [[["actor"], ["published", "desc"]]], likes: [ [["object"], ["published", "desc"]], [["actor"], ["object"]], ], } ``` ```js [/db/triggers.js] export default { likes: [ { key: "inc_likes", on: "create", fn: [ ["update()", [{ likes: { _$: ["inc"] } }, "notes", "$after.object"]], ], }, ], } ``` ### Test Write test files under `/test`. ```js [/test/main.test.js] import assert from "assert" import { describe, it } from "node:test" import { acc } from "wao/test" import { DB } from "wdb-sdk" import { init } from "./utils.js" const actor1 = acc[1] const actor2 = acc[2] describe("Jots", () => { it.only("should post notes", async () => { const { id, db, q: mem } = await init() const a1 = new DB({ jwk: actor1.jwk, id, mem }) const a2 = new DB({ jwk: actor2.jwk, id, mem }) const note1 = await a1.set("add:note", { content: "hello a1" }, "notes") assert(note1.success) const note2 = await a2.set("add:note", { content: "hello a2" }, "notes") assert(note2.success) const tl = await db.get("notes", ["published", "desc"]) assert(tl.length === 2) const object = tl[0].id const like1 = await a1.set("add:like", { object }, "likes") assert(like1.success) const like2 = await a2.set("add:like", { object }, "likes") assert(like2.success) const like3 = await a2.set("add:like", { object }, "likes") assert(!like3.success) assert.equal((await db.get("notes", ["likes", "desc"], 1))[0].likes, 2) }) }) ``` Run the tests. ```bash yarn test-all ``` ### Deploy ```bash yarn deploy --wallet .wallet.json --hb http://localhost:10001 --db http://localhost:6364 ``` ## wdb-core `wdb-core` provides the composable devices internally used to build data pipelines. ### Installation ```bash yarn add wdb-core ``` ### build() `build()` is a factory method to construct a data pipeline. ```js import { kv, build } from "wdb-core" import { signer } from "wdb-core/utils" import { init_query } from "wdb-core/preset" import { dev_normalize, dev_verify, dev_parse, dev_auth, dev_write, dev_read, } from "wdb-core/devs" function get({ state, msg }) { state.opcode = "get" state.query = ["get", ...msg] return arguments[0] } function cget({ state, msg }) { state.opcode = "cget" state.query = ["cget", ...msg] return arguments[0] } let store = {} const io = { put: async (key, val) => (store[key] = val), get: key => store[key] ?? null, transaction: async fn => fn(), } const wdb = build({ write: [dev_normalize, dev_verify, dev_parse, dev_auth, dev_write], read: [dev_normalize, dev_parse, dev_read], __read__: { get: [get, dev_parse, dev_read], cget: [cget, dev_parse, dev_read], }, }) const db = wdb(kv(io, c => {})) const sign = signer({ jwk: acc[0].jwk, id: "db-1" }) // initializing the database const res = await db.write(await sign("init", init_query)).val() // creating users directory await db.write(await sign( "set:dir", { schema: { type: "object", required: ["name", "age"] }, auth: [ ["set:user,add:user,update:user,upsert:user,del:user", [["allow()"]]], ], }, "_", "users", )) // adding Bob to users const Bob = { name: "Bob", age: 23 } await db.write(await sign("set:user", Bob, "users", "Bob")) console.log(await db.get("users").val()) // => [ Bob ] ``` ### kv() `kv()` is an adaptor function between in-memory store and the underlying kv storage. Using an in-memory store: ```js import { kv, build } from "wdb-core" import { open } from "lmdb" const wdb = build({ write: [dev_normalize, dev_verify, dev_parse, dev_auth, dev_write], read: [dev_normalize, dev_parse, dev_read], __read__: { get: [get, dev_parse, dev_read], cget: [cget, dev_parse, dev_read], }, }) let store = {} const io = { put: async (key, val) => (store[key] = val), get: key => store[key] ?? null, transaction: async (fn)=> fn() } const db = wdb(kv(io, c => { /* callback */ })) ``` Using LMDB: ```js import { kv, build } from "wdb-core" import { open } from "lmdb" const wdb = build({ write: [dev_normalize, dev_verify, dev_parse, dev_auth, dev_write], read: [dev_normalize, dev_parse, dev_read], __read__: { get: [get, dev_parse, dev_read], cget: [cget, dev_parse, dev_read], }, }) const io = open({ path: "./db" }) const db = wdb(kv(io, c => { /* callback */ })) ``` ### db() `db()` is a pre-built NoSQL database. ```js import { kv, db as wdb } from "wdb-core" import { open } from "lmdb" const io = open({ path: "./db" }) const db = wdb(kv(io, c => { /* callback */ })) ``` ### io() `io()` gives you an in-memory kv store with the same interface as `lmdb`, which can be passed to `kv()`. ```js import { kv, io, db as wdb } from "wdb-core" const mem_io = io() const db = wdb(kv(mem_io, c => { /* callback */ })) ``` ### queue() `queue()` wraps `db` to prevent conflicts providing a query queue mechanism. Without queue, it's hard to guarantee ACID to all requests. ```js import { kv, io, db as wdb, queue } from "wdb-core" const mem_io = io() const db = wdb(kv(mem_io, c => { /* callback */ })) const q = queue(db) ``` ### mem() `mem()` conveniently returns a set of in-memory components. ```js import { mem } from "wdb-core" const { q, db, io, kv } = mem() ``` ### sql() `sql()` is a pre-built SQL database using [SQLite](https://sqlite.org/). ```js import { kv, sql } from "wdb-core" import { open } from "lmdb" const io = open({ path: "./db" }) const db = sql(kv(io, c => { /* callback */ })) ``` ### vec() `vec()` is a pre-built vector database using [LanceDB](https://lancedb.com/). ```js import { kv, vec } from "wdb-core" import { open } from "lmdb" const io = open({ path: "./db" }) const db = vec(kv(io, c => { /* callback */ })) ``` :::warning `sql()` and `vec()` are highly experimental features that demonstrate building different types of databases. ::: ### Devices You can use the preset devices to build data pipelines for NoSQL DBs, RDBs, and experimental vector DBs. You caould easily swap one of the devices and build a custom pipeline. #### Common Devices * `dev_normalize` * `dev_verify` * `dev_auth` * `dev_read` #### NoSQL Devices The default NoSQL pipeline uses our own implementation of B+ tree indexers, query parsers, and planners. * `dev_parse` * `dev_write` ```js function get({ state, msg }) { state.opcode = "get" state.query = ["get", ...msg] return arguments[0] } function cget({ state, msg }) { state.opcode = "cget" state.query = ["cget", ...msg] return arguments[0] } export default build({ write: [dev_normalize, dev_verify, dev_parse, dev_auth, dev_write], read: [dev_normalize, dev_parse, dev_read], __read__: { get: [dev_get, dev_parse, dev_read], cget: [dev_cget, dev_parse, dev_read], }, }) ``` #### Relational Devices The default RDB pipeline uses [SQLite](https://www.sqlite.org/). * `dev_parse_vec` * `dev_write_vec` ```js function sql({ msg, env: { sql } }) { return sql.prepare(msg[0]).all() } export default build({ write: [dev_normalize, dev_verify, dev_parse, dev_write], __read__: { sql: [sql] }, }) ``` #### Vector Devices The default vector DB pipeline uses [LanceDB](https://lancedb.github.io/lancedb/) with models from [Hugging Face](https://huggingface.co/models). * `dev_parse_vec` * `dev_write_vec` ```js function search({ msg, env: { kv } }) { return kv.search(...msg) } function vectorSearch({ msg, env: { kv } }) { return kv.vectorSearch(...msg) } function query({ msg, env: { kv } }) { return kv.query(...msg) } export default build({ async: true, write: [dev_normalize, dev_verify, dev_parse, dev_write], __read__: { search: [search], vectorSearch: [vectorSearch], query: [query] }, }) ``` ## wdb-sdk ### Installation ```bash yarn add wdb-sdk ``` ### Instantiation ```js import { DB } from 'wdb-sdk' const db = new DB({ jwk, url, hb, id, mem }) ``` * `jwk` : signer Arweave wallet JWK * `url` : DB rollup server URL (default: `http://localhost:6364`)\` * `hb` : HyperBEAM WAL node URL (default: `http://localhost:10001`)\` * `id` : DB ID, don't specify it when spawning a new DB * `mem` : use in-memory DB from `wdb-core` :::info If `jwk` is unspecified in browser environments, [Wander](https://www.wander.app/) will be automatically used. ::: In-memory DB is useful for lightning-fast testing without a rollup server and HyperBEAM. ```js import { DB } from 'wdb-sdk' import { mem } from 'wdb-core' const { q, db, io, kv } = mem() // q (queue) wraps db to prevent conflicts const db = new DB({ jwk: owner.jwk, mem: q }) // q can be shared with multiple clients const user1 = new DB({ jwk: user1.jwk, mem: q }) const user2 = new DB({ jwk: user2.jwk, mem: q }) ``` #### ready() You can ensure the rollup server is available and ready when instantiating. ```js const db = await new DB({ jwk, url, hb }).ready() ``` `ready()` will ensure `${url}/status` returns `status="ok"`. If you are the HyperBEAM node operator, you can also start a rollup node by passing `true` to `ready()`. ```js const db = await new DB({ jwk, url, hb }).ready(true) ``` This will ensure `${hb}/~weavedb@1.0/start` returns `status=true`. ### spawn() Spawn a new DB instance with `db.spawn()`, which returns a DB ID. It spawns a new process to record WAL (Write-Ahead Logging) on the HyperBEAM node, then use the process ID to create a new DB instance on the Rollup node. The rollup node automatically bundles up queries and asyncronously dumps them to the HyperBEAM process in the background, while serving users at in-memory speed with cloud-level performance. ```js const id = await db.spawn() ``` ### mkdir() Creating a dir (directory) with `schema`, `auth`, and `name` definitions. `auth` defines custom query types and their rules such as `set:user` and `del:user`. ```js await db.mkdir({ name: "users", schema: { type: "object", required: ["name", "age"] }, auth: [["set:user,update:user,del:user", [["allow()"]]]], }) ``` ### set() `set()` executes write queries according to the `auth` rules and the `schema` set on the `dir`. ```js const res = await db.set("add:user", { name: "Bob", age: 25 }) const { success, error, result, query } = res ``` #### result `result` comes with variety of metadata. * `hashpath` : hash to track verifiable compute steps (AO-Core protocol) * `signer` : signer of the HTTP message * `msg` : HTTP message (HTTP message signature) * `nonce` : nonce to prevent replay attacks * `op` : operation ( `op` = `opcode` + `:` + `oprand` ) * `opcode` : operation type * `operand` : custom operation name * `query` : query without `op` * `dir` : directory to update * `before` : data before updated * `data` : data after updated * `id` : DB ID * `ts` : timestamp * `result` : contains transaction index and updated keys and data in the underlying kv store #### Query Types (`opcode`) There are 5 `opcode` types you can specify in `auth` with `mkdir()`. * `add` : add a doc with auto-generated docid, always add a new doc * `set` : add a new doc with specified docid, whether or not it exists * `update` : update a doc if it doesn't exist with docid, reject if it exists * `upsert` : add a doc with docid if it doesn't exist, update it if it exists * `del` : delete a doc with docid if it exists #### Special Modifiers (`_$`) `_$` provides special modifiers for field updates. ```js // deleting the field, this is different from assigning null await db.update({ name: {_$: "del" }}, "users", "Bob") // timestamp await db.update({ date: {_$: "ts" }}, "users", "Bob") // message signer await db.update({ signer: {_$: "signer" }}, "users", "Bob") ``` You can also execute advanced logic to modify the field by defining FPJSON in an array. ```js // increment await db.update({ age: {_$: ["inc"] }}, "users", "Bob") // add await db.update({ age: {_$: ["add", 5] }}, "users", "Bob") // remoive items await db.update({ favs: {_$: ["without", ["apple"]] }}, "users", "Bob") ``` ### batch() Batch-execute multiple write queries. ```js const Bob = { name: "Bob", age: 20, favs: [ "apple", "orange" ]} const Alice = { name: "Alice", age: 40, favs: [ "orange", "peach" ]} const Beth = { name: "Mike", age: 30, favs: [ "grapes", "orange" ]} const Mike = { name: "Mike", age: 30, favs: [ "peach", "apple" ]} await db.batch([ [ "update:user", { favs: Bob.favs }, "users", "Bob" ], [ "set:user", Alice, "users", "Alice" ], [ "set:user", Beth, "users", "Beth" ], [ "set:user", Mike, "users", "Mike" ] ]) ``` ### get() #### single doc ```js const Bob = await db.get("users", "Bob") ``` #### multiple docs ```js const users = await db.get("users") ``` #### sort ```js await db.get("users", ["age", "desc"]) // => [ Alice, Beth, Bob, Mike ] await db.get("users", ["name", "desc"]) // => [ Mike, Bob, Beth, Alice ] await db.get("users", ["age", "desc"], ["name", "desc"]) // => [ Alice, Mike, Beth, Bob ] ``` :::warning The last one with the multiple sort fields requires adding the index first. `await db.addIndex([["age", "desc"], ["name", "desc"]], "users")` ::: #### limit ```js await db.get("users", ["age", "desc"], 2) // => [ Alice, Beth ] ``` #### where `==` | `!=` | `>` | `>=` | `<` | `<=` | `in` | `not-in` | `array-contains` | `array-contains-any` ```js await db.get("users", ["age", "==", 30]) // => [ Beth, Mike ] await db.get("users", ["age", "!=", 30]) // => [ Bob, Alice ] await db.get("users", ["age", ">", 30]) // => [ Alice ] await db.get("users", ["age", ">=", 30]) // => [ Beth, Mike, Alice ] await db.get("users", ["age", "<", 30]) // => [ Bob ] await db.get("users", ["age", "<=", 30]) // => [ Bob, Beth, Mike ] await db.get("users", ["age", "in", [20, 30]]) // => [ Bob, Beth, Mike ] await db.get("users", ["age", "not-in", [20, 30]]) // => [ Alice ] await db.get("users", ["favs", "array-conteins", "apple"]) // => [ Bob, Mike ] await db.get("users", ["favs", "array-conteins-any", ["apple", "peach"]) // => [ Alice, Bob, Mike ] ``` #### skip `startAt` | `startAfter` | `endAt` | `endBefore` ```js await db.get("users", ["age", "asc"], ["startAt", 30]) // => [ Beth, Mike, Alice ] await db.get("users", ["age", "asc"], ["startAfter", 30]) // => [ Alice ] await db.get("users", ["age", "asc"], ["endAt", 30]) // => [ Bob, Beth, Mike ] await db.get("users", ["age", "asc"], ["endBefore", 30]) // => [ Bob ] ``` ### cget() `cget()` has the same interface as `get()` but it returns doc with metadata. ```js const { __cursor__, dir, id, data: Bob } = await db.cget("users", "Bob") ``` You can also use the result from `cget()` as a cursor with skip operations. ```js const cursor = await db.cget("users", "Bob") await db.get("users", ["age", "asc"], ["startAfter", cursor]) // => [ Beth, Mike, Alice ] ``` ### iter() `iter()` internally handles `cget()` and makes pagination easier. ```js let { docs, next, isNext } = await db.iter("users", ["age", "asc"], 2) // docs => [ { data: Bob }, { data: Beth } ] while(isNext) { ;({ docs, next, isNext } = await next()) // docs => [ { data: Mike }, { data: Alice } ] } ``` ### nonce() The current `nonce()` of the assigned signer. If the client has the wrong nonce, it auto-sync with the latest nonce and retry the failed query. ```js const nonce = await db.nonce() ``` ### stat() `stat(dir)` returns dir info including `schema`, `auth`, `indexes`, and `triggers`. `index` is the leaf position of the dir in the zk sparse merkle tree. `auth` is the FPJSON rules for authentication and data transformations. `autoid` is the auto-increment id by `add`, the actual dir ids are in base64 form. ```js const stat = await db.stat("users") // => { schema, auth, indexes, triggers, index, autoid } ``` ### addIndex() Multi-field sorting requires adding the index first. ```js await db.addIndex([["age", "desc"], ["name", "desc"]], "users") await db.get("users", ["age", "desc"], ["name", "desc"]) // => [ Alice, Mike, Beth, Bob ] ``` :::warning Only the database owner can add/remove indexes. ::: ### removeIndex() ```js await db.removeIndex([["age", "desc"], ["name", "desc"]], "users") await db.get("users", ["age", "desc"], ["name", "desc"]) // => Error ``` ### setSchema() Update the JSON schema for the dir. ```js await db.setSchema(schema, "users") ``` :::warning Only the database owner can set schemas. ::: ### setAuth() Update the auth rules for the dir. ```js await db.setAuth(auth, "users") ``` :::warning Only the database owner can set auth rules. ::: ### addTrigger() Add a trigger to the dir. ```js const trigger = { key: "inc_user_count", on: "create", fn: [ ["update()", [{ user_count: { _$: ["inc"] } }, "meta", "app_stats"]], ], } // increment user_count in meta/app_stats when a user is created await db.addTrigger(trigger, "users") ``` :::warning Only the database owner can add/remove triggers. ::: ### removeTrigger() Remove a trigger from the dir. Specify a `key` to remove. ```js await db.removeTrigger({ key: "inc_user_count" }, "users") ``` ### Utilities #### wdb23() Convert an Arweave address to a [WDB23](/specs/wdb23) address. ```js import { wdb23 } from "wdb-sdk" const addr23 = wdb23(arweave_address) ``` #### wdb160() Generate a [WDB160](/specs/wdb160) hash from multiple imputs. ```js import { wdb160 } from "wdb-sdk" const hash = wdb160(["abc", "def"]) ``` ## zkjson ### Installation ```bash yarn add zkjson ``` #### Encoder / Decoder Encode / Decode JSON ```javascript const { encode, decode, toSignal, fromSignal } = require("zkjson") const json = { a : 1 } const encoded = encode(json) // [ 1, 1, 97, 2, 1, 0, 1 ] const signal = toSignal(encoded) // [ '111129712111011' ] const encoded2 = fromSignal(signal) // [ 1, 1, 97, 2, 1, 0, 1 ] const decoded = decode(encoded2) // { a : 1 } ``` Encode / Decode paths ```javascript const { toSignal, fromSignal, encodePath, decodePath, path } = require("zkjson") const _path = "a" const encodedPath = encodePath(_path) // [ 1, 1, 97 ] const signalPath = toSignal(encodedPath) // [ "1111297" ] const encodedPath2 = fromSignal(signalPath) // [ 1, 1, 97 ] const decodedPath = decodePath(encodedPath) // "a" const signalPath2 = path(_path) // [ "1111297" ] ``` Encode / Decode values ```javascript const { toSignal, fromSignal, encodeVal, decodeVal, val } = require("zkjson") const _val = 1 const encodedVal = encodeVal(_val) // [ 2, 1, 0, 1 ] const signalVal = toSignal(encodedVal) // [ "12111011" ] const encodedVal2 = fromSignal(signalVal) // [ 2, 1, 0, 1 ] const decodedVal = decodeVal(encodedVal) // 1 const signalVal2 = val(_val) // [ "12111011" ] ``` Encode / Decode conditional queries ```javascript const { toSignal, fromSignal, encodeQuery, decodeQuery, query } = require("zkjson") const _query = [ "$gt", 1 ] const encodedQuery = encodeQuery(_query) // [ 12, 2, 1, 0, 1 ] const signalQuery = toSignal(encodedQuery) // [ "21212111011" ] const encodedQuery2 = fromSignal(signalQuery) // [ 12, 2, 1, 0, 1 ] const decodedQuery = decodeQuery(encodedQuery) // [ "$gt", 1 ] const signalQuery2 = query(_query) // [ "21212111011" ] ``` #### Document ID \<> Index Conversion ```javascript const { toIndex, fromIndexs } = require("zkjson") const index = toIndex("zkJSON") // 1513609181413 const str = fromIndex(index) // "zkJSON" ``` #### Doc ```javascript const { loadFixture } = require("@nomicfoundation/hardhat-network-helpers") const { path, Doc } = require("../../sdk") const { resolve } = require("path") const { expect } = require("chai") async function deploy() { const Verifier = await ethers.getContractFactory("Groth16VerifierJSON") const verifier = await Verifier.deploy() const MyApp = await ethers.getContractFactory("SimpleJSON") const myapp = await MyApp.deploy(verifier.address) return { myapp } } describe("MyApp", function () { let myapp this.timeout(0) beforeEach(async () => { const dep = await loadFixture(deploy) myapp = dep.myapp }) it("should verify JSON", async function () { const doc = new Doc({ wasm: resolve( __dirname, "../../circom/build/circuits/json/index_js/index.wasm" ), zkey: resolve( __dirname, "../../circom/build/circuits/json/index_0001.zkey" ), }) const json = { num: 1, float: 1.23, str: "string", bool: true, null: null, array: [1, 2, 3], } // query number const zkp = await doc.genProof({ json, path: "num" }) expect((await myapp.qInt(path("num"), zkp)).toNumber()).to.eql(1) // query string const zkp2 = await doc.genProof({ json, path: "str" }) expect(await myapp.qString(path("str"), zkp2)).to.eql("string") // query bool const zkp3 = await doc.genProof({ json, path: "bool" }) expect(await myapp.qBool(path("bool"), zkp3)).to.eql(true) // query null const zkp4 = await doc.genProof({ json, path: "null" }) expect(await myapp.qNull(path("null"), zkp4)).to.eql(true) // query float const zkp5 = await doc.genProof({ json, path: "float" }) expect( (await myapp.qFloat(path("float"), zkp5)).map(f => f.toNumber()) ).to.eql([1, 2, 123]) // query array and get number const zkp6 = await doc.genProof({ json, path: "array" }) expect( (await myapp.qCustom(path("array"), path("[1]"), zkp6)).toNumber() ).to.eql(2) // conditional operator const zkp7 = await doc.genProof({ json, path: "num", query: ["$gt", 0] }) expect(await myapp.qCond(path("num"), zkp7.slice(15, 21), zkp7)).to.eql( true ) }) }) ``` #### DB ```javascript const { loadFixture } = require("@nomicfoundation/hardhat-network-helpers") const { toIndex, path, DB } = require("../../sdk") const { resolve } = require("path") const { expect } = require("chai") async function deploy() { const [committer] = await ethers.getSigners() const VerifierRU = await ethers.getContractFactory("Groth16VerifierRU") const verifierRU = await VerifierRU.deploy() const VerifierDB = await ethers.getContractFactory("Groth16VerifierDB") const verifierDB = await VerifierDB.deploy() const MyRU = await ethers.getContractFactory("SimpleRU") const myru = await MyRU.deploy( verifierRU.address, verifierDB.address, committer.address ) return { myru, committer } } describe("MyRollup", function () { let myru, committer, db, col_id, ru this.timeout(0) beforeEach(async () => { const dep = await loadFixture(deploy) myru = dep.myru committer = dep.committer }) it("should verify rollup transactions", async function () { db = new DB({ level: 100, size_path: 5, size_val: 5, size_json: 256, size_txs: 10, level_col: 8, wasmRU: resolve( __dirname, "../../circom/build/circuits/rollup/index_js/index.wasm" ), zkeyRU: resolve( __dirname, "../../circom/build/circuits/rollup/index_0001.zkey" ), wasm: resolve( __dirname, "../../circom/build/circuits/db/index_js/index.wasm" ), zkey: resolve( __dirname, "../../circom/build/circuits/db/index_0001.zkey" ), }) await db.init() col_id = await db.addCollection() const people = [ { name: "Bob", age: 10 }, { name: "Alice", age: 20 }, { name: "Mike", age: 30 }, { name: "Beth", age: 40 }, ] let txs = people.map(v => { return [col_id, v.name, v] }) const zkp = await db.genRollupProof(txs) await myru.commit(zkp) const zkp2 = await db.genProof({ json: people[0], col_id, path: "age", id: "Bob", }) expect( ( await myru.qInt([col_id, toIndex("Bob"), ...path("age")], zkp2) ).toNumber() ).to.eql(10) const zkp3 = await db.genProof({ json: people[3], col_id, path: "name", id: "Beth", }) expect( await myru.qString([col_id, toIndex("Beth"), ...path("name")], zkp3) ).to.eql("Beth") }) }) ``` ### ZK Circuits There are 5 main circuits, and each circuit is built on top of the preceding one. #### Circuits ##### JSON.circom The base building block to prove JSON with an efficient encoding. * `size_json` : JSON size : default `256` * `size_path` : path size : default `4` * `size_val` : value size : default `8` ##### Collection.circom A collection proven by a sparse merkle tree (SMT) can contain many JSON documents (2 \*\* 168 by default). * `level` : collection SMT level : default `168` * `size_json` : JSON size : default `256` * `size_path` : path size : default `4` * `size_val` : value size : default `8` ##### DB.circom A database proven by a sparse merkle tree (SMT) can contain many collections (2 \*\* 8 by default). * `level_col` : DB SMT level : default `8` * `level` : collection SMT level : default `168` * `size_json` : JSON size : default `256` * `size_path` : path size : default `4` * `size_val` : value size : default `8` ##### Query.circom Query proves a JSON data insert or update by a single write query. * `level_col` : DB SMT level : default `8` * `level` : collection SMT level : default `168` * `size_json` : JSON size : default `256` ##### Rollup.circom Rollup proves batch data transitions. * `tx_size` : max number of queries in a batch : default `10` * `level_col` : DB SMT level : default `8` * `level` : collection SMT level : default `168` * `size_json` : JSON size : default `256` #### Powers of Tau The first thing you need to do is to set up a powers of tau by a ceremony. As the power goes up the generation time and the wasm file size increases exponentially, and what power required for each circuit depends on the parameters above. So you need to find the right balance with the parameters of each circuit for your application. For instance, `power 20` required for the default `Rollup` circuit settings takes hours with a normal consumer computer. To run a ceremony, ```bash yarn ceremony --power 14 ``` Generated files are located at `build/pot`. You can also specify `entropy` and `name` for the ceremony. Refer to [the Circom docs](https://docs.circom.io/getting-started/proving-circuits/) for what they mean. ```bash yarn ceremony --power 14 --name "first contribution" --entropy "some random value" ``` The same goes with the compiling process below. #### Compile Circuit You can specify the parameters when compiling a circuit. Unspecified parameters will use the default values. For instance, to compile the `JSON` circuit, ```bash yarn compile --power 14 --circuit json --size_json 256 --size_path 4 --size_val 8 ``` To compile the `Rollup` circuit, you might need to increase `--max-old-space-size` of NodeJS. ```bash yarn compile --power 20 --circuit rollup --tx_size 10 --level_col 8 --level 168 --size_json 256 ``` All the generated files are stored at `build/circuits` including a Solidity verifier contract. #### Concept of Some Parameters ##### size The base unit of `size` is `uint`. Circom by default uses the module of `21888242871839275222246405745257275088548364400416034343698204186575808495617` (77 digits) and Solidity's base storage block is `uint256` and allows 78 digits. So zkJSON efficiently encodes JSON and packs it into blocks of 76 digits, which is one `uint`. `path_size=5` means, 5 \* 76 digits are allowed for the query path when encoded, and it will be represented within `uint[5]` in Solidity. on the Solidity side, however, zkJSON uses dynamic arrays `uint[]`, so it will be more space-efficient than the max set size. But the zk-circuits cannot prove data sizes more than the set size. The default `json_size` is set `256`, which is 256 \* 76 digits and should be sufficient for most JSON data. ##### level `level` is the level of the sparse merkle tree (SMT). As the litepaper describes, the level of SMT for Collection determines how many alphanumeric characters each document ID can contain. It's determined by ```math Number of Characters = \frac{\log_{10}(2^{\text{Level}})}{2} ``` `level=168` can allow 28 characters in document ID. This is significant because document IDs are often used in access control rules of NoSQL databases (with WeaveDB, for instance). 28 characters can fit compressed Ethereum addresses (20 bytes) in Bse64 format. For DB, `level_col` determines how many collections the DB can contain. The collection IDs use the direct index numbers and are not converted to an alphanumeric representation, so `level_col=8` (2 \*\* 8 = 256) collections should be sufficient for most applications. But you are free to set a different value. #### Default Parameters and Required POT | Circuit | POT | size\_json | size\_path | size\_val | level | level\_col | tx\_size | | -------------- | --- | ---------- | ---------- | --------- | ----- | ---------- | -------- | | **JSON** | 14 | 256 | 4 | 8 | | | | | **Collection** | 16 | 256 | 4 | 8 | 168 | | | | **DB** | 16 | 256 | 4 | 8 | 168 | 8 | | | **Query** | 17 | 256 | | | 168 | 8 | | | **Rollup** | 20 | 256 | | | 168 | 8 | 10 | **Currently the SDK only works with `size_json=256` due to some hash logic. Keep it 256 for now please.** ### Solidity Contracts #### ZKQuery.sol ```solidity interface ZKQuery { function toArr(uint[] memory json) internal pure returns (uint[] memory); function _qNull (uint[] memory path, uint[] memory zkp) internal pure returns (bool); function _qBool (uint[] memory path, uint[] memory zkp) internal pure returns (bool); function _qInt (uint[] memory path, uint[] memory zkp) internal pure returns (int); function _qFloat (uint[] memory path, uint[] memory zkp) internal pure returns (uint[3] memory); function _qString (uint[] memory path, uint[] memory zkp) internal pure returns (string memory); function _qRaw (uint[] memory path, uint[] memory zkp) internal pure returns (uint[] memory); function getNull (uint[] memory path, uint[] memory raw) internal pure returns (bool); function getBool (uint[] memory path, uint[] memory raw) internal pure returns (bool); function getInt (uint[] memory path, uint[] memory raw) internal pure returns (int); function getFloat (uint[] memory path, uint[] memory raw) internal pure returns (uint[3] memory); function getString (uint[] memory path, uint[] memory raw) internal pure returns (string memory); } ``` #### ZKJson.sol ```solidity interface ZKJSON { function _validateQueryJSON( uint[] memory path, uint[] memory zkp, uint size_path, uint size_val ) internal pure returns (uint[] memory); } ``` #### ZKRollup.sol ```solidity interface ZKRollup { function _validateQueryRU( uint[] memory path, uint[] memory zkp, uint size_path, uint size_val ) internal view returns (uint[] memory); function commit (uint[] memory zkp) public returns (uint); } ``` #### Examples `ZKJson` and `ZKRollup` inherit `ZKQuery`. You need to either inherit `ZKJson` or `ZKRollup` to build your own ZKDB-enabled contract. You can install `zkjson` node package and use the contract located at `node_modules/zkjson/contracts` in your Solidity contract. To install the package, ```bash yarn add zkjson ``` ##### Simple zkJSON ```solidity // SPDX-License-Identifier: MIT pragma solidity >=0.7.0 <0.9.0; import "ZKJson.sol"; interface VerifierJSON { function verifyProof(uint[2] calldata _pA, uint[2][2] calldata _pB, uint[2] calldata _pC, uint[12] calldata _pubSignals) view external returns (bool); } contract SimpleJSON is ZKJson { uint constant SIZE_PATH = 5; uint constant SIZE_VAL = 5; constructor (address _verifierJSON){ verifierJSON = _verifierJSON; } function verify(uint[] memory zkp) private view returns (bool) { uint[SIZE_PATH + SIZE_VAL + 2] memory sigs; ( uint[2] memory _pA, uint[2][2] memory _pB, uint[2] memory _pC, uint[] memory _sigs ) = _parseZKP(zkp); for(uint i = 0; i < sigs.length; i++) sigs[i] = _sigs[i]; require(VerifierJSON(verifierJSON).verifyProof(_pA, _pB, _pC, sigs), "invalid proof"); return true; } function validateQuery(uint[] memory path, uint[] memory zkp) private view returns(uint[] memory){ verify(zkp); return _validateQueryJSON(path, zkp, SIZE_PATH, SIZE_VAL); } function qInt (uint[] memory path, uint[] memory zkp) public view returns (int) { uint[] memory value = validateQuery(path, zkp); return _qInt(value); } function qFloat (uint[] memory path, uint[] memory zkp) public view returns (uint[3] memory) { uint[] memory value = validateQuery(path, zkp); return _qFloat(value); } function qRaw (uint[] memory path, uint[] memory zkp) public view returns (uint[] memory) { uint[] memory value = validateQuery(path, zkp); return _qRaw(value); } function qString (uint[] memory path, uint[] memory zkp) public view returns (string memory) { uint[] memory value = validateQuery(path, zkp); return _qString(value); } function qBool (uint[] memory path, uint[] memory zkp) public view returns (bool) { uint[] memory value = validateQuery(path, zkp); return _qBool(value); } function qNull (uint[] memory path, uint[] memory zkp) public view returns (bool) { uint[] memory value = validateQuery(path, zkp); return _qNull(value); } function qCond (uint[] memory path, uint[] memory cond, uint[] memory zkp) public view returns (bool) { uint[] memory value = validateQuery(path, zkp); return _qCond(value, cond); } function qCustom (uint[] memory path, uint[] memory path2, uint[] memory zkp) public view returns (int) { uint[] memory value = validateQuery(path, zkp); return getInt(path2, value); } } ``` ##### Simple zkRollup ```solidity // SPDX-License-Identifier: MIT pragma solidity >=0.7.0 <0.9.0; import "ZKRollup.sol"; interface VerifierDB { function verifyProof(uint[2] calldata _pA, uint[2][2] calldata _pB, uint[2] calldata _pC, uint[14] calldata _pubSignals) view external returns (bool); } contract SimpleRU is ZKRollup { uint constant SIZE_PATH = 5; uint constant SIZE_VAL = 5; address public verifierDB; constructor (address _verifierRU, address _verifierDB, address _committer){ verifierRU = _verifierRU; verifierDB = _verifierDB; committer = _committer; } function verify(uint[] memory zkp) private view returns (bool) { uint[SIZE_PATH + SIZE_VAL + 4] memory sigs; ( uint[2] memory _pA, uint[2][2] memory _pB, uint[2] memory _pC, uint[] memory _sigs ) = _parseZKP(zkp); for(uint i = 0; i < sigs.length; i++) sigs[i] = _sigs[i]; require(VerifierDB(verifierDB).verifyProof(_pA, _pB, _pC, sigs), "invalid proof"); return true; } function validateQuery(uint[] memory path, uint[] memory zkp) private view returns(uint[] memory){ verify(zkp); return _validateQueryRU(path, zkp, SIZE_PATH, SIZE_VAL); } function qInt (uint[] memory path, uint[] memory zkp) public view returns (int) { uint[] memory value = validateQuery(path, zkp); return _qInt(value); } function qFloat (uint[] memory path, uint[] memory zkp) public view returns (uint[3] memory) { uint[] memory value = validateQuery(path, zkp); return _qFloat(value); } function qRaw (uint[] memory path, uint[] memory zkp) public view returns (uint[] memory) { uint[] memory value = validateQuery(path, zkp); return _qRaw(value); } function qString (uint[] memory path, uint[] memory zkp) public view returns (string memory) { uint[] memory value = validateQuery(path, zkp); return _qString(value); } function qBool (uint[] memory path, uint[] memory zkp) public view returns (bool) { uint[] memory value = validateQuery(path, zkp); return _qBool(value); } function qCond (uint[] memory path, uint[] memory cond, uint[] memory zkp) public view returns (bool) { uint[] memory value = validateQuery(path, zkp); return _qCond(value, cond); } function qNull (uint[] memory path, uint[] memory zkp) public view returns (bool) { uint[] memory value = validateQuery(path, zkp); return _qNull(value); } } ``` ## Advanced Start with ZK In this advanced tutorial, you will build a database for a social dapp with zk circuits, and query it from Ethereum, as well as AO. create a db project using the `web-cli create` command. ```bash npx wdb-cli create mydb && cd mydb yarn add wdb-sdk ``` ### Define Database To keep it simple, we will only make one `dir` called `posts`, and allow `add:post`. ```js [/db/schema.js] export default { posts: { type: "object", required: ["uid", "body", "date"], properties: { uid: { type: "string", pattern: "^[a-zA-Z0-9_-]{43}$" }, body: { type: "string", minLength: 1, maxLength: 240 }, date: { type: "integer", minimum: 0, maximum: 9999999999999 }, }, }, } ``` ```js [/db/auth.js] export default { posts: [ [ "add:post", [ ["mod()", { uid: "$signer", date: "$ts" }], // add uid and date ["allow()"], // allow anyone ], ], ], } ``` ```js [/db/indexes.js] export default {} ``` ```js [/db/triggers.js] export default {} ``` Make sure you are running a local rollup node and a HyperBEAM node, then have `.wallet.json` in the app root directory. Let's deploy the DB. ```bash yarn deploy --wallet .wallet.json ``` ### Frontend Dapp We are going to build the simplest social app ever using NextJS! For simplicity, use the old `pages` structure insted of `apps`. ```bash npx create-next-app myapp && cd myapp ``` :::code-group ```jsx [/pages/index.js] import { useRef, useEffect, useState } from "react" import { DB } from "wdb-sdk" export default function Home() { const [posts, setPosts] = useState([]) const [body, setBody] = useState("") const db = useRef() const getPosts = async () => { setPosts(await db.current.get("posts", ["date", "desc"], 10)) } useEffect(() => { void (async () => { db.current = new DB({ id: YOUR_DB_ID }) await getPosts() })() }, []) return ( <>