Skip to content

GraphQL-Pouch: Local Data with a Schema

An open-source experiment from the summer of 2016: a GraphQL runtime that generates a complete CRUD API on top of PouchDB from a schema shorthand – including synchronization via the CouchDB replication protocol.

≈ 10 min readSoftwareGraphQLAPI

GraphQL-Pouch: Local Data with a Schema

type Post implements Node {
  id: ID!
  rev: String
  title: String
  body: String
  published: Boolean
}

Five fields, one interface, not a single line of resolver code. This file is the complete backend of a small blog system – data storage, HTTP endpoint and searchable API documentation included. You pass it to graphql-pouch, and a few seconds later a running GraphQL API with generated queries and mutations answers at http://127.0.0.1:3000/graphql/blog: post, allPosts, upsertPost, deletePost. Data isn’t stored on a remote database server but locally in PouchDB – and if you want, the process synchronizes its documents with other nodes via the CouchDB replication protocol.

That’s the short version of GraphQL-Pouch, an open-source project I published on GitHub and npm this month. It is explicitly an experiment. I didn’t want to know GraphQL only from blog posts but from the inside – and for years, my best way to get to know a technology from the inside has been the same: build something small and real with it and use it until the uncomfortable spots become visible.

One year of GraphQL, many open questions#

For context: the GraphQL specification has been public for just one year. Facebook released it in July 2015 together with graphql-js as the reference implementation; internally, the technology has been running there since 2012. The tooling around it is still modest: Relay as a client framework, GraphiQL as an interactive UI for exploring an API, express-graphql as the HTTP binding for Node.js. If you build a GraphQL server today, you write type definitions and resolvers by hand against graphql-js – one function per field that says where the data comes from.

By the third resolver I wrote myself, I noticed how mechanical this work is for simple cases. A Post type with five fields, a document database behind it: read by key, list all documents, write, delete. It’s always the same pattern, and patterns that are always the same can be generated by a machine. The really interesting work lies in the schema – in the question of which types, fields and relationships the application needs. That became the thesis of the experiment: if the schema is the only source, how far does a runtime get that generates the rest?

The second thread came from a different direction. In several frontend projects, I’ve seen backend development become the bottleneck: the team waits for endpoints, for database migrations, for deployments – and in the meantime builds mock-ups that later have to be painstakingly removed. A backend that emerges from a schema file and runs locally without infrastructure reverses this order. The frontend team defines the data contract itself, works against a real API immediately and negotiates later what a “real” backend has to take over.

From shorthand to a running API#

GraphQL-Pouch reads the schema shorthand – the same terse notation the GraphQL documentation uses for its examples – and builds a complete, executable schema from it at runtime with graphql-js. For every type, this produces generated root fields including resolvers against PouchDB. The parser for this goes back to Matthew Mueller’s project graph.ql, whose groundwork saved me a lot of time.

The Relay conventions are active by default: a Node interface with global IDs, connections with edges and node for lists, clientMutationId in the mutation payloads. That was a deliberate decision, because Relay is currently the most demanding GraphQL client, and if you meet its conventions, you meet the simpler cases along the way. If you don’t want them, you turn them off with --no-relay and get plain lists.

A query against the generated blog schema looks like this:

query BlogFrontpage {
  allPosts {
    edges {
      node {
        id
        rev
        title
        published
      }
    }
  }
}

mutation PublishPost {
  upsertPost(input: {
    id: "hello-graphql",
    title: "GraphQL over local data",
    published: true,
    clientMutationId: "1"
  }) {
    post {
      id
      rev
    }
  }
}

In development mode (graphql-pouch --development), the server automatically serves GraphiQL at the endpoint. That sounds like a sideshow, but in day-to-day work it was the single biggest gain: the API can be explored interactively from the first minute, with autocompletion and built-in documentation from the schema comments. No Postman, no fiddling with cURL – type in a query, see the result.

Beyond that, the runtime brings a few things that turned the experiment, almost unnoticed, into a usable tool:

  • several schemas in parallel under their own URL paths (/graphql/blog, /graphql/cms)
  • JWT authentication via a startup parameter, including a CLI command for signing tokens
  • serving static files, so a React application can live right next to it
  • migrating schema files, custom functions and static content via the command line

Why PouchDB sits underneath#

The choice of database was the real architecture decision of the project. PouchDB is a JavaScript implementation of the CouchDB ideas: document-oriented, revision-based, and – this is the crucial point – with the CouchDB replication protocol as a built-in synchronization mechanism. In the Node.js process, PouchDB stores data directly in the file system via LevelDB. There is no database server, no connection string, no migrations. npm install -g graphql-pouch, start, done – the state lives as files next to the process.

For a tool meant to decouple frontend development from the backend, this self-sufficiency is not a convenience feature but the core. Every infrastructure dependency a developer first has to install is a hurdle before the first query.

It gets interesting as soon as more than one node is involved. With graphql-pouch --remote http://user:pass@couch.example.com, the local PouchDB replicates against a remote CouchDB – in both directions. Several instances behind a load balancer share the same data this way, without any of them being the “master.” The price is known and well documented in the CouchDB world: there is no single source of truth, only eventual consistency. Every node is available at all times, but not every node sees the same state at all times.

flowchart LR
  SDL[Schema file<br/>shorthand] --> RT[graphql-pouch<br/>runtime]
  RT --> GEN[Generated fields<br/>allPosts, upsertPost, deletePost]
  UI[GraphiQL /<br/>Relay client] --> GEN
  GEN --> P[(PouchDB<br/>LevelDB in the file system)]
  P <-->|CouchDB<br/>replication protocol| C[(CouchDB<br/>remote node)]

That I treat this property not as a flaw but as a feature has to do with the target audience: frontends. A UI that waits for confirmation from a remote server feels sluggish. With its optimistic updates, Relay already comes with a model that shows changes immediately and reconciles them with the server response later. Eventual consistency at the data level and optimistic updates at the client level are two expressions of the same idea – the architecture of GraphQL-Pouch takes this idea seriously instead of hiding it.

The revision is in the schema – on purpose#

If you read the Post example above carefully, you’ll stumble over the rev field. That’s the document revision from the CouchDB world, and you can ask with good reason whether such a storage detail should be visible in an API. I decided it should, after a concrete failed attempt: in an early version, I handled the revision internally and simply passed writes through. That works exactly as long as nobody writes at the same time. As soon as two clients change the same document – or a replicated node plays back an older version – the last one silently overwrites the first. The lost state only gets noticed when a user misses it, and by then it can no longer be reconstructed.

With a visible revision, this flips around. An upsertPost with an outdated revision fails in a controlled way, because PouchDB detects the conflict:

sequenceDiagram
  participant A as Client A
  participant B as Client B
  participant API as graphql-pouch
  participant DB as PouchDB
  A->>API: upsertPost(id, rev: "1-abc", ...)
  API->>DB: put(document)
  DB-->>API: ok, rev: "2-def"
  B->>API: upsertPost(id, rev: "1-abc", ...)
  API->>DB: put(document)
  DB-->>API: 409 conflict
  API-->>B: error instead of silent overwrite

The client then has to read again and decide what happens to the competing changes. That’s less convenient than a schema without rev, but the inconvenience is honest: it shows a problem at the place where it arises instead of pushing it into the future. With local, replicated data, the write conflict is not an edge case you can abstract away – it’s the normal case that just rarely happens.

When generated CRUD is no longer enough#

Generated endpoints get you surprisingly far, but not arbitrarily far. At some point, an application needs a query with domain logic, validation before writing or data from an external source. For these cases, GraphQL-Pouch has an escape hatch: custom functions in JavaScript that register under the name of a schema field and replace the generated resolver. Combined with use as an npm module, this can be embedded directly into an existing Express application – the companion repository graphql-pouch-as-library shows exactly this setup:

const express = require('express');
const expressGraphQL = require('express-graphql');
const graphqlPouch = require('graphql-pouch');

const schemaDefinition = `
type Post implements Node {
  id: ID!
  rev: String
  title: String
  body: String
  published: Boolean
}

type Query {
  postsByAuthor(author: String!): [Post]
}
`;

// custom resolver replaces the generated CRUD implementation
const customFunctions = {
  postsByAuthor: (ctx, input) => {
    legacyBlogService
      .fetchPostsByAuthor(input.author)
      .then(posts => ctx.success(posts));
  },
};

const mySchema = graphqlPouch.schema('blog', schemaDefinition, true, customFunctions);

const app = express();
app.use('/graphql', expressGraphQL({
  schema: mySchema.schema,
  graphiql: true,
}));
app.listen(3000);

This pattern turned out to be the most important finding of the experiment. Generation covers the mechanical part, and the custom functions take over the domain part – without having to change the foundation. GraphQL shows its strength as a facade here: behind the same schema, a generated PouchDB access, a call to an existing REST service and a hand-written calculation can exist side by side, and no client notices the difference.

What the experiment can’t do#

Being honest about an experiment includes the list of loose ends, and for GraphQL-Pouch it isn’t short:

  • Events are missing. PouchDB provides a changes feed that could drive notifications to clients – that’s on the roadmap but not built.
  • A batching and caching mechanism for resolvers is missing too; nested queries cause more database accesses than necessary.
  • The JWT support authenticates but doesn’t authorize – there are no roles or field permissions.
  • Generated CRUD stays generic: upsertPost says nothing about whether a post is being published, withdrawn or edited. A good domain API needs named operations, and those still have to be designed by a person.

The last point is the most important. Generation takes the typing work off your hands, not the design work. A schema that consists only of types with CRUD is a database with a prettier protocol – useful for prototypes, demos and the early frontend phase, but no substitute for a well-thought-out interface.

What I take away from this#

After a few weeks with my own tool, three observations remain. First: the approach of writing the schema first and deriving everything else from it changes conversations. You can negotiate concretely with frontend colleagues over a schema file with comments long before any server exists – the schema becomes the contract, not documentation after the fact. Second: local data with synchronization shifts complexity; it doesn’t eliminate it. What you save in infrastructure comes back as thinking about revisions and conflicts in the data model, and it’s better if the API makes that explicit. Third: a weekend experiment that you seriously build to the end – with a CLI, tests, documentation and publication on npm – teaches you more about a technology than any introduction, because it forces you to touch the edges too: error messages, authentication, operation with several nodes.

In the summer of 2016, I can’t judge whether GraphQL will catch on. But the idea of putting a typed data contract between client and server and aligning both sides with it seems more viable to me than much of what I’ve negotiated with REST conventions in ten years of API development. For that reason alone, the experiment was worth it.

Further reading#

Revised and translated from German with the help of AI.