(release): v1.1.0-rc1 (#238)

This lets the viewer application watch the file, or have the user manually refresh the view rather than opening a new browser tab.

CHANGELOG.md

@@ -1,5 +1,30 @@
 # Changelog
 
+## 1.0.0-rc1 (06-03-2020)
+
+This is a pre-release before the push to MELPA. It contains large
+internal changes, with little user-facing changes. Most notably, the
+backing storage has been changed to a SQLite database, and a
+templating system using `org-capture` is introduced.
+
+### Breaking Changes
+* [#200][gh-200] Move Org-roam cache into a SQLite database.
+* [#203][gh-203] Roam protocol is deprecated, in favour of extending org-roam-protocol.
+
+### New Features
+* [#182][gh-182] Support file name aliases via `#+ROAM_ALIAS`.
+* [#216][gh-216] Adds templating functionality by extending org-capture.
+* [#232][gh-232] Adds a prefix key to `org-roam-show-graph`, to generate graph without opening it.
+* [#233][gh-233] Adds `org-roam-graph-exclude-matcher`, which allows exclusion of nodes from graph.
+
+### Bugfixes
+* [#207][gh-207], [#221][gh-221] small bugfixes to Org-roam graph generation
+* [#230][gh-230] remove nonspacing marks from filenames, to prevent cross-platform errors
+
+### New Contributors
+* [@acowley][https://github.com/acowley]
+* [@teesloane][https://github.com/teesloane]
+
 ## 0.1.2 (2020-02-21)
 
 ### Breaking Changes
@@ -87,6 +112,13 @@ Mostly a documentation/cleanup release.
 [gh-141]: https://github.com/jethrokuan/org-roam/pull/141
 [gh-142]: https://github.com/jethrokuan/org-roam/pull/142
 [gh-143]: https://github.com/jethrokuan/org-roam/pull/143
+[gh-182]: https://github.com/jethrokuan/org-roam/pull/182
+[gh-188]: https://github.com/jethrokuan/org-roam/pull/188
+[gh-200]: https://github.com/jethrokuan/org-roam/pull/200
+[gh-207]: https://github.com/jethrokuan/org-roam/pull/207
+[gh-216]: https://github.com/jethrokuan/org-roam/pull/216
+[gh-221]: https://github.com/jethrokuan/org-roam/pull/221
+[gh-230]: https://github.com/jethrokuan/org-roam/pull/230
 
  # Local Variables:
  # eval: (auto-fill-mode -1)

README.md

@@ -28,10 +28,9 @@ As of February 2020, it is in a very early stage of development.
 
 Here's a screenshot of `org-roam`. The `org-roam` buffer shows
 backlinks for the active org buffer in the left window, as well as the
-surrounding content in the backlink file. The backlink database is
-built asynchronously in the background, and is not noticeable to the
-end user. The graph is generated from the link structure, and can be
-used to navigate to the respective files.
+surrounding content in the backlink file. The database is built once,
+and updated incrementally. The graph is generated from the link
+structure, and can be used to navigate to the respective files.
 
 ![img](doc/images/org-roam-graph.gif)
 
@@ -41,7 +40,6 @@ The recommended method is using use-package and straight, or a similar package m
 
 ```emacs-lisp
 (use-package org-roam
-      :after org
       :hook 
       (after-init . org-roam-mode)
       :straight (:host github :repo "jethrokuan/org-roam" :branch "develop")
@@ -59,7 +57,7 @@ For more detailed installation instructions (including instructions for
 Spacemacs users), please see [the installation
 documentation](https://org-roam.readthedocs.io/en/develop/installation/).
 
-## Knowledge Bases using Org-Roam
+## Knowledge Bases using Org-roam
 
 - [Jethro Kuan](https://braindump.jethro.dev/)
   ([Source](https://github.com/jethrokuan/braindump/tree/master/org))

doc/anatomy.md

@@ -0,0 +1,40 @@
+The bulk of Org-roam's functionality is built on top of vanilla
+Org-mode. However, to support additional functionality, Org-roam adds
+several Org-roam-specific keywords. These functionality are not
+crucial to effective use of Org-roam.
+
+## File Aliases
+
+Suppose you want a note to be referred to by different names (e.g.
+"World War 2", "WWII"). You may specify such aliases using the
+`#+ROAM_ALIAS` attribute:
+
+```org
+#+TITLE: World War 2
+#+ROAM_ALIAS: "WWII" "World War II"
+```
+
+
+## File Refs
+
+Refs are unique identifiers for files. Each note can only have 1 ref.
+For example, a note for a website may contain a ref:
+
+```org
+#+TITLE: Google
+#+ROAM_KEY: https://www.google.com/
+```
+
+These keys come in useful for when taking website notes, using the
+`roam-ref` protocol (see [Roam Protocol](roam_protocol.md)).
+
+Alternatively, add a ref for notes for a specific paper, using its
+[org-ref](https://github.com/jkitchin/org-ref) citation key:
+
+```org
+#+TITLE: Neural Ordinary Differential Equations
+#+ROAM_KEY: cite:chen18_neural_ordin_differ_equat
+```
+
+In future, the backlinks buffer will also show notes with that
+citation key.

doc/comparison.md

@@ -1,7 +1,7 @@
 ---
-title: "Comparing Org-Roam With Other Packages"
-metaTitle: "Comparing Org-Roam With Other Packages"
-metaDescription: "Comparing Org-Roam With Other Packages"
+title: "Comparing Org-roam With Other Packages"
+metaTitle: "Comparing Org-roam With Other Packages"
+metaDescription: "Comparing Org-roam With Other Packages"
 ---
 
 # Org-brain

doc/configuration.md

@@ -1,19 +1,13 @@
-To ensure that Org-roam remains manageable, the number of
-configuration options is deliberately kept small. However, we have
-attempted to accommodate as many usage styles as possible.
-
-In this section, we'll go over the main customization options
-available to Org-Roam. This section is *crucial*. We need to exploit
-the flexibility of Emacs, and mould our tools exactly to our liking.
+The number of configuration options is deliberately kept small, to
+keep the Org-roam codebase manageable. However, we attempt to
+accommodate as many usage styles as possible.
 
 All of Org-roam's customization options can be viewed via `M-x
 customize-group org-roam`.
 
 ## Setting the Org-roam Directory
 
-Perhaps the single most important variable to set is
-`org-roam-directory`. Set `org-roam-directory` to the folder
-containing all your Org files:
+Set `org-roam-directory` to the folder containing all your Org files:
 
 ```emacs-lisp
 (setq org-roam-directory "/path/to/org/")
@@ -22,6 +16,24 @@ containing all your Org files:
 Every Org file, at any level of nesting, within `/path/to/org/` is
 considered part of the Org-roam ecosystem.
 
+### Having More Than One Org-roam Directory
+
+Emacs supports directory-local variables, allowing the value of
+`org-roam-directory` to be different in different directories. It does
+this by checking for a file named `.dir-locals.el`. 
+
+To add support for multiple directories, override the
+`org-roam-directory` variable using directory-local variables. This is
+what `.dir-locals.el` may contain:
+
+```emacs-lisp
+((nil . ((org-roam-directory . "/path/to/here/"))))
+```
+
+All files within that directory will be treated as their own separate
+set of Org-roam files. Remember to run `org-roam-build-cache` from a
+file within that directory, at least once.
+
 ## Org-roam Buffer
 
 The Org-roam buffer defaults to popping up from the right. You may
@@ -40,65 +52,38 @@ frame width. For example:
 ```
 
 Will result in the Org-roam buffer taking up 40% of the screen width.
-I have found this to be a good number.
 
 ## Org-roam Links
 
 By default, links are inserted with the title as the link description.
-This can make them hard to distinguish from external links. If you
-wish, you may choose add special indicators for Org-roam links by
-tweaking `org-roam-link-title-format`, for example:
+This can make them hard to distinguish from external links. You may
+choose add special indicators for Org-roam links by tweaking
+`org-roam-link-title-format`, for example:
 
 ```emacs-lisp
 (setq org-roam-link-title-format "R:%s")
 ```
 
+If your version of Org is at least `9.2`, you may also choose to
+simply style the link differently, by customizing `org-roam-link-face`
+(`M-x customize-face org-roam-link`).
+
 ## Org-roam Files
 
-These customization options revolve around the Org files created and
-managed by Org-roam.
-
-### Automatically Creating Files Using Timestamp
-
-A common hassle is ensuring that files are uniquely named within the
-Org-roam directory. Org-roam's default workflow utilizes the title of
-Org files in all of its main commands (`org-roam-insert`,
-`org-roam-find-file`). Hence, having any unique file name is a decent
-option, and the default workflow uses the timestamp as the filename.
-
-The format of the filename is controlled by the function
-`org-roam-file-name-function`, which defaults to a format like
-`YYYYMMDDHHMMSS_title_here.org`. You may choose to define your own
-function to change this.
-
-If you wish to be prompted to change the file name on creation, set
-`org-roam-filename-noconfirm` to `nil`:
-
-```emacs-lisp
-(setq org-roam-filename-noconfirm nil)
-```
-
-It is then the user's responsibility to ensure that the file names are
-unique.
-
-### Autopopulating Titles
-
-The default workflow uses the title of the Org file in several
-commands. The title is specified via the `#+TITLE:` attribute,
-typically near the top of the file. The option
-`org-roam-autopopulate-title` defaults to `t`. When true, the title
-attribute is automatically inserted into the files created via
-Org-roam commands. Setting it to `nil` will disable this behaviour.
+Org-roam files are created and prefilled using Org-roam's templating
+system. The templating system is customizable, and the system is
+described in detail in the [Org-roam Template](templating.md) page.
 
 ### Encryption
 
 Encryption (via GPG) can be enabled for all new files by setting
 `org-roam-encrypt-files` to `t`. When enabled, new files are created
-with the .org.gpg extension and decryption are handled automatically
-by EasyPG. Note that this causes Emacs to ask for password when the
-cache is built (if you have an encrypted file in `org-roam-directory`)
-as well as each time a new file is created. It might be a good idea to
-cache the password in order to make this more managable.
+with the `.org.gpg` extension and decryption are handled automatically
+by EasyPG. 
+
+Note that Emacs will prompt for a password for encrypted files during
+cache updates if it requires reading the encrypted file. To reduce the
+number of password prompts, you may wish to cache the password.
 
 ## Org-roam Graph Viewer
 
@@ -107,13 +92,13 @@ Org-roam generates an SVG image using
 [Graph Setup](graph_setup.md) page.
 
 Org-roam tries its best to locate the Graphviz executable from your
-PATH, but if it fails to do so, you may set it manually:
+`PATH`, but if it fails to do so, you may set it manually:
 
 ```
 (setq org-roam-graphviz-executable "/path/to/dot")
 ```
 
-Org-roam also attempts to use Firefox (located on PATH) to view the
+Org-roam also attempts to use Firefox (located on `PATH`) to view the
 SVG, you may choose to set it to any compatible program:
 
 ```

doc/ecosystem.md

@@ -117,6 +117,12 @@ within Org-mode.
 [Org-ref][org-ref] does citation and bibliography management in
 Org-mode, and a great tool for scientific notes.
 
+### Spaced Repetition
+
+[Org-fc][org-fc] is a spaced repetition system that scales well with a
+large number of files. Other alternatives include
+[org-drill][org-drill], and [pamparam][pamparam].
+
 [deft]: https://jblevins.org/projects/deft/
 [notdeft]: https://github.com/hasu/notdeft
 [org-download]: https://github.com/abo-abo/org-download
@@ -124,3 +130,6 @@ Org-mode, and a great tool for scientific notes.
 [org-noter]: https://github.com/weirdNox/org-noter
 [interleave]: https://github.com/rudolfochrist/interleave
 [org-ref]: https://github.com/jkitchin/org-ref
+[org-fc]: https://github.com/l3kn/org-fc/
+[org-drill]: https://orgmode.org/worg/org-contrib/org-drill.html
+[pamparam]: https://github.com/abo-abo/pamparam

doc/graph_setup.md

@@ -1,78 +0,0 @@
-The setup is similar to that of org-protocol. Here `roam://` links are
-defined, and need to be associated with an application. 
-
-The gist of the setup is setting up a Bash script to trim off the
-`roam://` prefix from the link, causing the desktop application to
-call `emacsclient path/to/org-roam-file.org`.
-
-## Linux
-
-Create a desktop application. I place mine in
-`~/.local/share/applications/roam.desktop`:
-
-```
-[Desktop Entry]
-Name=Org-Roam Client
-Exec=/home/jethro/.local/bin/launch_emacs %u
-Icon=emacs-icon
-Type=Application
-Terminal=false
-MimeType=x-scheme-handler/roam
-```
-
-Note the `Exec` key is set to a bash script poorly named
-`launch_emacs`. You can set it to whatever you want.
-
-Create the corresponding bash script, and make it executable. Here's
-how it looks like:
-
-```bash
-#!/usr/bin/env bash
-emacsclient "${1#*:}"
-```
-
-Finally, associate `roam://` links with the desktop application by
-running in your shell:
-
-```bash
-xdg-mime default roam.desktop x-scheme-handler/roam
-```
-
-## Mac OS
-
-One solution to this, recommended in [Issue
-#115](https://github.com/jethrokuan/org-roam/issues/115), is to use
-[Platypus](https://github.com/sveinbjornt/Platypus). Here are the
-instructions for setting up with Platypus and Chrome:
-
-1. Create an executable `launch-emacs.sh` script:
-
-```sh
-#!/usr/bin/env bash
-/usr/local/bin/emacsclient --no-wait "${1#*:}"
-```
-
-2. Install and launch Platypus (with [Homebrew](https://brew.sh/)):
-
-```sh
-brew cask install playtpus
-```
-
-3. Playtpus settings:
-
-- App Name: `OrgRoam`
-- Script Type: `env` and `/usr/bin/env`
-- Script Path: `/path/to/your/launch-emacs.sh`
-- Tick Accept dropped items and click Settings
-- Tick Accept dropped files
-- Tick Register as URI scheme handler
-- Add `roam` as a protocol
-- Create the app
-
-To disable the "confirm" prompt in Chrome, you can also make Chrome
-show a checkbox to tick, so that the `OrgRoam` app will be used
-without confirmation. To do this, run in a shell:
-
-```sh
-defaults write com.google.Chrome ExternalProtocolDialogShowAlwaysOpenCheckbox -bool true
-```

doc/index.md

@@ -1,40 +1,62 @@
 ![org-roam][org-roam-intro-image]
 
-## What is Org-Roam?
+## What is Org-roam?
 
-Org-roam is a rudimentary [Roam][roamresearch] replica built around
-the all-powerful [Org-mode][org]. 
+Org-roam is a [Roam][roamresearch] replica built around the
+all-powerful [Org-mode][org].
 
-Like Roam, Org-roam offers a powerful and effortless non-hierarchical
-note-taking approach. With Org-roam, notes flow naturally, making
-note-taking fun and easy. To understand more about Roam, I recommend
-the following links:
+Org-roam is a solution for effortless non-hierarchical note-taking,
+building upon Org-mode. This solution is heavily inspired by [Roam
+Research][roamresearch]. With Org-roam, notes flow naturally, making
+note-taking fun and easy. Org-roam should also work as a plug-and-play
+solution for anyone already using Org-mode for their personal wiki.
 
--   [Building a second brain in
-    Roam](https://reddit.com/r/RoamResearch/comments/eho7de/building_a_second_brain_in_roamand_why_you_might)
--   [Roam: Why I Love It and How I Use
-    It](https://www.nateliason.com/blog/roam)
+To understand more about Roam, a collection of links are available in
+[the appendix](notetaking_workflow.md).
 
-The goal of the project is to implement core features of Roam around
-Org-mode, and eventually introduce features enabled by the Emacs
-ecosystem. 
+Org-roam aims to implement the core features of Roam, leveraging the
+mature ecosystem around Org-mode where possible. Eventually, we hope
+to further introduce features enabled by the Emacs ecosystem.
 
-## Why build Org-Roam?
+## Why use Org-roam?
 
-With Org-roam, you:
+##### Private and Secure
 
-1. Never have to leave Emacs.
-2. Can leverage all the powerful features of Org-mode: LaTeX, tables,
-   and the whole to-do ecosystem (org-agenda etc.)
-3. Be in full control your second brain, and access it offline. Never
-   share your data with anyone
+Edit your personal wiki completely offline, entirely in your control.
+Encrypt your notes with GPG.
+
+##### Longevity
+
+Unlike web solutions like Roam research, the notes are first and
+foremost plain Org-mode files -- Org-roam simply builds up an
+auxilliary database to give the personal wiki superpowers. Having your
+notes in plain-text is crucial for the longevity of your wiki. Never
+have to worry about proprietary web solutions being taken down. Edit
+your plain-text notes in notepad if all other editors cease to exist.
+
+##### Free and Open-source
+
+Org-roam is free and open-source, which means that if you feel unhappy
+with any part of Org-roam, you may choose to extend Org-roam, or open
+a PR.
+
+##### Leverage the Org-mode Ecosystem
+
+Over the years, Emacs and Org-mode has developed into a mature system
+for plain-text organization. Building upon Org-mode already puts
+Org-roam light-years ahead of many other solutions.
+
+Emacs is also a fantastic interface for editing text, and we can
+inherit many of the powerful text-navigation and editing packages
+available to Emacs.
 
 There are several packages that are similar to Org-roam, see the
 [Comparison](comparison.md) page for a detailed comparison.
 
 ## Project Status
 
-As of February 2020, it is in a very early stage of development. 
+As of March 2020, most of the core functionality and interfaces have
+stabilized, and a stable release is near.
 
 [org-roam-intro-image]: images/org-roam-intro.png
 [roamresearch]: https://www.roamresearch.com/

doc/installation.md

@@ -3,9 +3,8 @@
 The recommended method is using [use-package][use-package] and
 [straight][straight], or a similar package manager.
 
-```
+```emacs-lisp
 (use-package org-roam
-      :after org
       :hook 
       (after-init . org-roam-mode)
       :straight (:host github :repo "jethrokuan/org-roam" :branch "develop")
@@ -14,6 +13,7 @@ The recommended method is using [use-package][use-package] and
       :bind (:map org-roam-mode-map
               (("C-c n l" . org-roam)
                ("C-c n f" . org-roam-find-file)
+               ("C-c n b" . org-roam-switch-to-buffer)
                ("C-c n g" . org-roam-show-graph))
               :map org-mode-map
               (("C-c n i" . org-roam-insert))))
@@ -26,47 +26,54 @@ directory and add it to your load path:
 git clone https://github.com/jethrokuan/org-roam/ ~/.emacs.d/elisp/org-roam
 ```
 
-```
+```emacs-lisp
 (use-package org-roam
-      :after org
-      :load-path "elisp/"
-      :hook 
-      (after-init . org-roam-mode)
-      :straight (:host github :repo "jethrokuan/org-roam" :branch "develop")
-      :custom
-      (org-roam-directory "/path/to/org-files/")
-      :bind (:map org-roam-mode-map
+  :load-path "elisp/"
+  :hook
+  (after-init . org-roam-mode)
+  :straight (:host github :repo "jethrokuan/org-roam" :branch "develop")
+  :custom
+  (org-roam-directory "/path/to/org-files/")
+  :bind (:map org-roam-mode-map
               (("C-c n l" . org-roam)
                ("C-c n f" . org-roam-find-file)
+               ("C-c n b" . org-roam-switch-to-buffer)
                ("C-c n g" . org-roam-show-graph))
               :map org-mode-map
               (("C-c n i" . org-roam-insert))))
 ```
 
-Or without use-package:
+Or without `use-package`:
 
-```
+```emacs-lisp
 (add-to-list 'load-path "./elisp")
 (require 'org-roam)
+(define-key 'org-roam-mode-map (kbd "C-c n l") #'org-roam)
+(define-key 'org-roam-mode-map (kbd "C-c n f") #'org-roam-find-file)
+(define-key 'org-roam-mode-map (kbd "C-c n b") #'org-roam-switch-to-buffer)
+(define-key 'org-roam-mode-map (kbd "C-c n g") #'org-roam-switch-to-buffer)
+(define-key 'org-mode-map (kbd "C-c n i") #'org-roam-insert)
+(org-roam-mode +1)
 ```
 
-There are a number of important configuration options, that greatly
-affect the Roam workflow. Do look through them at the
-[Configuration](configuration.md) page.
+The [Configuration](configuration.md) page details some of the common
+configuration options available.
 
 [use-package]: https://github.com/jwiegley/use-package
 [straight]: https://github.com/raxod502/straight.el
 
 ## Spacemacs
-If you are using Spacemacs, you can easily install org-roam by creating a simple layer that wraps org-roam. Paste the following into a new file `/.emacs.d/private/org-roam/packages.el`.
-```
+If you are using Spacemacs, install org-roam by creating a simple
+layer that wraps Org-roam. Paste the following into a new file
+`~/.emacs.d/private/org-roam/packages.el`.
+
+```emacs-lisp
 (defconst org-roam-packages
   '((org-roam :location
         (recipe :fetcher github :repo "jethrokuan/org-roam" :branch "develop"))))
 
 (defun org-roam/init-org-roam ()
     (use-package org-roam
-        :after org
         :hook
         (after-init . org-roam-mode)
         :custom
@@ -90,4 +97,37 @@ If you are using Spacemacs, you can easily install org-roam by creating a simple
             "rg" 'org-roam-show-graph))))
 ```
 
-Next, append `org-roam` to the `dotspacemacs-configuration-layers` list in your `.spacemacs` configuration file. Reload (`SPC f e R`) or restart Emacs to load `org-roam`. It's functions are available under the prefix `SPC a r` and `, r` when visiting an org-mode buffer. 
+Next, append `org-roam` to the `dotspacemacs-configuration-layers`
+list in your `.spacemacs` configuration file. Reload (`SPC f e R`) or
+restart Emacs to load `org-roam`. It's functions are available under
+the prefix `SPC a r` and `, r` when visiting an org-mode buffer.
+
+## Doom Emacs
+
+If you are using [Doom Emacs](https://github.com/hlissner/doom-emacs), configure packages as explained in the [getting started](https://github.com/hlissner/doom-emacs/blob/develop/docs/getting_started.org#configuring-packages) guide. 
+
+Declare Org-roam as a package in your `~/.doom.d/packages.el`:
+
+```elisp
+;; ~/.doom.d/packages.el
+
+(package! org-roam
+  :recipe (:host github :repo "jethrokuan/org-roam" :branch "develop"))
+```
+
+Subsequently, in your `~/.doom.d/config.el` file, configure Org-roam:
+
+```elisp
+;; ~/.doom.d/config.el
+(use-package! org-roam
+  :commands (org-roam-insert org-roam-find-file org-roam)
+  :init 
+  (setq org-roam-directory "/path/to/org-files/")
+  (map! :leader 
+        :prefix "n"
+        :desc "Org-Roam-Insert" "i" #'org-roam-insert
+        :desc "Org-Roam-Find"   "/" #'org-roam-find-file
+        :desc "Org-Roam-Buffer" "r" #'org-roam)
+  :config
+  (org-roam-mode +1))
+```

doc/notetaking_workflow.md

@@ -8,6 +8,9 @@
 - [Roam: Why I Love It and How I Use It][5]
 - [Adam Keesling's Twitter Thread][6]
 
+## Threads
+- [Ask HN: How to Take Good Notes][8]
+
 ## What to Do With Your Notes
 - [How to Use Roam to Outline a New Article in Under 20 Minutes][2]
 
@@ -18,3 +21,4 @@
 [5]: https://www.nateliason.com/blog/roam
 [6]: https://twitter.com/adam_keesling/status/1196864424725774336?s=20
 [7]: https://blog.jethro.dev/posts/how_to_take_smart_notes_org/
+[8]: https://news.ycombinator.com/item?id=22473209

doc/roam_protocol.md

@@ -0,0 +1,161 @@
+## What is Roam protocol?
+
+Org-roam extending `org-protocol` with 2 protocols: the `roam-file`
+and `roam-ref` protocol.
+
+## The `roam-file` protocol
+
+This is a simple protocol that opens the path specified by the `file`
+key (e.g. `org-protocol://roam-file?file=/tmp/file.org`). This is used
+in the generated graph.
+
+## The `roam-ref` Protocol
+
+This protocol finds or creates a new note with a given `ROAM_KEY` (see
+[Anatomy](anatomy.md)):
+
+![roam-ref](images/roam-ref.gif)
+
+To use this, create a Firefox bookmarklet as follows:
+
+```javascript
+javascript:location.href =
+'org-protocol:/roam-ref?template=r&ref='
++ encodeURIComponent(location.href)
++ '&title='
++ encodeURIComponent(document.title)
+```
+
+where `template` is the template key for a template in
+`org-roam-ref-capture-templates`. More documentation on the templating
+system can be found [here](templating.md).
+
+These templates should contain a `#+ROAM_KEY: ${ref}` in it.
+
+## Setting up Org-roam protocol
+
+To enable org-roam's protocol extensions, you have to add the
+following to your init file:
+
+```emacs-lisp
+(require 'org-roam-protocol)
+```
+
+The instructions for setting up `org-protocol` can be found
+[here][org-protocol-inst], but they are reproduced below.
+
+We will also need to create a desktop application for `emacsclient`.
+The instructions for various platforms are shown below:
+
+## Linux
+
+Create a desktop application. I place mine in
+`~/.local/share/applications/org-protocol.desktop`:
+
+```
+[Desktop Entry]
+Name=Org-Protocol
+Exec=emacsclient %u
+Icon=emacs-icon
+Type=Application
+Terminal=false
+MimeType=x-scheme-handler/org-protocol
+```
+
+Associate `org-protocol://` links with the desktop application by
+running in your shell:
+
+```bash
+xdg-mime default org-protocol.desktop x-scheme-handler/org-protocol
+```
+
+To disable the "confirm" prompt in Chrome, you can also make Chrome
+show a checkbox to tick, so that the `Org-Protocol Client` app will be used
+without confirmation. To do this, run in a shell:
+
+```sh
+sudo mkdir -p /etc/opt/chrome/policies/managed/
+sudo tee /etc/opt/chrome/policies/managed/external_protocol_dialog.json >/dev/null <<'EOF'
+{
+  "ExternalProtocolDialogShowAlwaysOpenCheckbox": true
+}
+EOF
+sudo chmod 644 /etc/opt/chrome/policies/managed/external_protocol_dialog.json
+```
+
+and then restart Chrome (for example, by navigating to <chrome://restart>) to
+make the new policy take effect.
+
+See [here](https://www.chromium.org/administrators/linux-quick-start)
+for more info on the `/etc/opt/chrome/policies/managed` directory and
+[here](https://cloud.google.com/docs/chrome-enterprise/policies/?policy=ExternalProtocolDialogShowAlwaysOpenCheckbox)
+for information on the `ExternalProtocolDialogShowAlwaysOpenCheckbox`
+policy.
+
+
+## Mac OS
+
+One solution is to use
+[Platypus](https://github.com/sveinbjornt/Platypus). Here are the
+instructions for setting up with Platypus and Chrome:
+
+1. Install and launch Platypus (with [Homebrew](https://brew.sh/)):
+
+```sh
+brew cask install playtpus
+```
+2. Create a script `launch_emacs.sh`:
+
+```
+#!/usr/bin/env bash
+/usr/local/bin/emacsclient --no-wait $1
+```
+
+3. Create a Platypus app with the following settings:
+
+```
+| Setting                        | Value                     |
+|--------------------------------+---------------------------|
+| App Name                       | "OrgProtocol"             |
+| Script Type                    | "env" · "/usr/bin/env"    |
+| Script Path                    | "path/to/launch-emacs.sh" |
+| Interface                      | None                      |
+| Accept dropped items           | true                      |
+| Remain running after execution | false                     |
+```
+
+Inside `Settings`:
+
+```
+| Setting                        | Value          |
+|--------------------------------+----------------|
+| Accept dropped files           | true           |
+| Register as URI scheme handler | true           |
+| Protocol                       | "org-protocol" |
+```
+
+To disable the "confirm" prompt in Chrome, you can also make Chrome
+show a checkbox to tick, so that the `OrgProtocol` app will be used
+without confirmation. To do this, run in a shell:
+
+```sh
+defaults write com.google.Chrome ExternalProtocolDialogShowAlwaysOpenCheckbox -bool true
+```
+
+
+##### Note for Emacs Mac Port
+
+If you're using [Emacs Mac Port](https://github.com/railwaycat/homebrew-emacsmacport), it
+registered its `Emacs.app` as the default handler for the URL scheme
+`org-protocol`. We have to make our `OrgProtocol.app` the default
+handler instead (replace `org.yourusername.OrgProtocol` with your app
+identifier):
+
+```
+$ defaults write com.apple.LaunchServices/com.apple.launchservices.secure LSHandlers -array-add \
+'{"LSHandlerPreferredVersions" = { "LSHandlerRoleAll" = "-"; }; LSHandlerRoleAll = "org.yourusername.OrgProtocol"; LSHandlerURLScheme = "org-protocol";}'
+```
+
+Then restart your computer.
+
+[org-protocol-inst]: https://orgmode.org/worg/org-contrib/org-protocol.html

doc/templating.md

@@ -0,0 +1,83 @@
+Rather than creating blank files on `org-roam-insert` and
+`org-roam-find-file`, it is may be desirable to prefill the file with
+content. This may include:
+
+- Time of creation
+- File it was created from
+- Clipboard content
+- Any other data you may want to input manually
+
+This requires a complex template insertion system, but fortunately,
+Org ships with a powerful one: `org-capture`. However, org-capture was
+not designed for such use. Org-roam abuses `org-capture` to some
+extent, extending its syntax. To first understand how org-roam's
+templating system works, it may be useful to look into org-capture.
+
+## Org-roam Templates
+
+The org-roam capture template extends org-capture's template with 2
+additional properties:
+
+1. `:file-name`: This is the file name template used when a new note
+   is created.
+2. `:head`: This is the template that is inserted on initial note
+   creation. 
+
+### Org-roam Template Expansion
+
+Org-roam's template definitions also extend org-capture's template
+syntax, to allow prefilling of strings. In many scenarios,
+`org-roam-capture` is passed a mapping between variables and strings.
+For example, during `org-roam-insert`, a title is prompted for. If the
+title doesn't already exist, we would like to create a new file,
+without prompting for the title again.
+
+Variables passed are expanded with the `${var}` syntax. For example,
+eduring `org-roam-insert`, `${title}` is prefilled for expansion. Any
+variables that do not contain strings, are prompted for values using
+`completing-read`.
+
+After doing this expansion, the org-capture's template expansion
+system is used to fill up the rest of the template. You may read up
+more on this on [org-capture's documentation
+page](https://orgmode.org/manual/Template-expansion.html#Template-expansion).
+
+For example, take the template: `"%<%Y%m%d%H%M%S>-${title}"`, with the title
+`"Foo"`. The template is first expanded into `%<%Y%m%d%H%M%S>-Foo`. Then
+org-capture expands `%<%Y%m%d%H%M%S>` with timestamp: e.g.
+`20200213032037-Foo`.
+
+This templating system is used throughout org-roam templates.
+
+### Template examples
+
+Here I walkthrough the default template, reproduced below.
+
+```
+("d" "default" plain (function org-roam--capture-get-point)
+     "%?"
+     :file-name "%<%Y%m%d%H%M%S>-${slug}"
+     :head "#+TITLE: ${title}\n"
+     :unnarrowed t)
+```
+
+1. The template has short key `"d"`. If you have only one template,
+   org-roam automatically chooses this template for you.
+2. The template is given a description of `"default"`.
+3. `plain` text is inserted. Other options include Org headings via
+   `entry`.
+4. `(function org-roam--capture-get-point)` should not be changed.
+5. `"%?"` is the template inserted on each call to `org-roam-capture`.
+   This template means don't insert any content, but place the cursor
+   here.
+6. `:file-name` is the file-name template for a new note, if it
+doesn't yet exist. This creates a file at path that looks like
+   `/path/to/org-roam-directory/20200213032037-foo.org`.
+7. `:head` contains the initial template to be inserted (once only),
+   at the beginning of the file. Here, the title global attribute is
+   inserted.
+8. `:unnarrowed t` tells org-capture to show the contents for the
+   whole file, rather than narrowing to just the entry.
+
+Other options you may want to learn about include `:immediate-finish`.
+

doc/tour.md

@@ -1,69 +1,65 @@
 Org-roam was built to support a workflow that was not possible with
 vanilla Org-mode. This flow is modelled after the [Zettelkasten
 method][zettelkasten], and many of [Roam Research][roam]'s workflows.
-Understanding this flow is crucial! Org-roam doesn't auto-magically
-make your note-taking better -- it's changing the note-taking workflow
-that does.
+It is crucial to understand that Org-roam does not auto-magically make
+note-taking better -- it's changing the note-taking workflow that
+does.
 
-To understand more the methods and madness, the [Note-Taking
-Workflow][appendix:ntw] page contains a page of useful references.
-I've also written [a post][jethro-blog-post] about how I use Org-roam.
+To understand more about the methods and madness, the [Note-Taking
+Workflow][appendix:ntw] page contains a page of useful references. The
+author has also written [a post][jethro-blog-post] about how he uses
+Org-roam.
 
-Without further ado, let's begin!
+## Activating Org-roam
 
-## Building the Cache
+Org-roam's entry point is the global minor `org-roam-mode`. This sets
+up Emacs with several hooks, for keeping the org-roam cache
+consistently updated, as well as showing the backlinks buffer. 
 
-Assuming you've set `org-roam-directory` appropriately, running `M-x
-org-roam--build-cache-async` should build up the caches that will
-allow you to begin using Org-roam. I do this on startup:
-
-```emacs-lisp
-(add-hook 'after-init-hook 'org-roam--build-cache-async)
-```
+The cache is a sqlite database named `org-roam.db`, which resides at
+the root of the `org-roam-directory`. Activating `org-roam-mode`
+builds the cache, which may take a while the first time, but is
+generally instantaneous in subsequent runs. To build the cache
+manually again, run `M-x org-roam-build-cache`.
 
 ## Finding a Note
 
-`org-roam-find-file` shows you the list of notes you currently have in
-Org-roam. Selecting the title will bring you to the corresponding
-note. Entering a title of a note that does not yet exist will create a
-new note with that title.
+`org-roam-find-file` shows the list of titles for notes that reside in
+`org-roam-directory`. Selecting a note title will bring you to the
+corresponding note. Entering a title of a note that does not yet exist
+will create a new note with that title.
 
 ![org-roam-find-file](images/org-roam-find-file.gif)
 
 ## Inserting Links
 
-Within your Org-roam notes, you are encouraged to liberally insert
-links to existing (or new) Org-roam notes with `org-roam-insert`.
-Entering a non-existent title will also create a new note with that
-title.
+`org-roam-insert` insert links to existing (or new) notes. Entering a
+non-existent title will also create a new note with that title.
 
 ![org-roam-insert](images/org-roam-insert-filetag.gif)
 
-It is crucial for good usage of Org-roam to insert links liberally,
-where you want them the notes to resurface!
+Good usage of Org-roam requires liberally linking files. This allows
+the build-up of a dense knowledge graph.
 
 ## The Org-roam Buffer
 
-All of Org-roam's operations are designed such that the built cache is
-a consistent view of the inter-connectivity between your notes. The
-Org-roam buffer shows backlinks: i.e. the files that link to the
-currently viewed file, along with some surrounding context. The
-Org-roam buffer will always show the backlinks for the current
-Org-roam file in view.
+The Org-roam buffer is often displayed in the side window. It shows
+backlinks for the currently active Org-roam note, along with some
+surrounding context.
 
 ![org-roam-buffer](images/org-roam-buffer.gif)
 
 ## Exporting the Graph
 
-It's also possible to export the links as a graph, using graphviz. The
-generated graph is navigable in Emacs, but requires some additional
-setup, which I describe in the [Graph Appendix][appendix:graph-setup]
-page.
+Org-roam also uses Graphviz to generate a graph, with notes as nodes,
+and links between them as edges. The generated graph can be used to
+navigate to the files, but this requires some additional setup
+described in the [Roam Protocol][appendix:roam-protocol] page.
 
 ![org-roam-graph](images/org-roam-graph.gif)
 
 [zettelkasten]: https://zettelkasten.de/
 [appendix:ntw]: notetaking_workflow.md
-[appendix:graph-setup]: graph_setup.md
+[appendix:roam-protocol]: roam_protocol.md
 [roam]: https://www.roamresearch.com/
 [jethro-blog-post]: https://blog.jethro.dev/posts/how_to_take_smart_notes_org/

mkdocs.yml

@@ -1,17 +1,19 @@
-site_name: "Org-Roam: Roam + Org-Mode = ♥"
+site_name: "Org-roam"
 repo_url: https://github.com/jethrokuan/org-roam/
 edit_uri: edit/master/doc/
 copyright: "Copyright (C) 2020 Jethro Kuan and contributors"
 docs_dir: doc
 nav:
 - Home: index.md
-- A Tour of Org-Roam: tour.md
+- A Tour of Org-roam: tour.md
 - Installation: installation.md
 - Configuration: configuration.md
+- Anatomy of an Org-roam file: anatomy.md
+- The Templating System: templating.md
 - Ecosystem: ecosystem.md
 - Similar Packages: comparison.md
 - "Appendix: Note-taking Workflow": notetaking_workflow.md
-- "Appendix: Graph Setup": graph_setup.md
+- "Appendix: Roam Protocol": roam_protocol.md
 markdown_extensions:
   - admonition
   - pymdownx.betterem:

org-roam-protocol.el

@@ -0,0 +1,103 @@
+;;; org-roam-protocol.el --- Protocol handler for roam:// links  -*- coding: utf-8; lexical-binding: t -*-
+
+;; Copyright © 2020 Jethro Kuan <jethrokuan95@gmail.com>
+;; Author: Jethro Kuan <jethrokuan95@gmail.com>
+;; URL: https://github.com/jethrokuan/org-roam
+;; Keywords: org-mode, roam, convenience
+;; Version: 1.0.0-rc1
+;; Package-Requires: ((emacs "26.1") (org "9.0"))
+
+;; This file is NOT part of GNU Emacs.
+
+;; This program is free software; you can redistribute it and/or modify
+;; it under the terms of the GNU General Public License as published by
+;; the Free Software Foundation; either version 3, or (at your option)
+;; any later version.
+;;
+;; This program is distributed in the hope that it will be useful,
+;; but WITHOUT ANY WARRANTY; without even the implied warranty of
+;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;; GNU General Public License for more details.
+;;
+;; You should have received a copy of the GNU General Public License
+;; along with GNU Emacs; see the file COPYING.  If not, write to the
+;; Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor,
+;; Boston, MA 02110-1301, USA.
+
+;;; Commentary:
+;;
+;; We extend org-protocol, adding custom Org-roam handlers. The setup
+;; instructions for `org-protocol' can be found in org-protocol.el.
+;;
+;; We define 2 protocols:
+;;
+;; 1. "roam-file": This protocol simply opens the file given by the FILE key
+;; 2. "roam-ref": This protocol creates or opens a note with the given REF
+;;
+;;; Code:
+
+(require 'org-protocol)
+(require 'org-roam)
+
+(declare-function org-roam-find-ref "org-roam" (&optional info))
+(declare-function org-roam--capture-get-point "org-roam" ())
+
+(defvar org-roam-ref-capture-templates
+  '(("r" "ref" plain (function org-roam--capture-get-point)
+     ""
+     :file-name "${slug}"
+     :head "#+TITLE: ${title}
+#+ROAM_KEY: ${ref}\n"
+     :unnarrowed t))
+  "The Org-roam templates used during a capture from the roam-ref protocol.
+Details on how to specify for the template is given in `org-roam-capture-templates'.")
+
+(defun org-roam-protocol-open-ref (info)
+  "Process an org-protocol://roam-ref?ref= style url with INFO.
+
+It opens or creates a note with the given ref.
+
+  javascript:location.href = \\='org-protocol://roam-ref?template=r&ref=\\='+ \\
+        encodeURIComponent(location.href) + \\='&title=\\=' \\
+        encodeURIComponent(document.title) + \\='&body=\\=' + \\
+        encodeURIComponent(window.getSelection())"
+  (when-let* ((alist (org-roam--plist-to-alist info))
+              (decoded-alist (mapcar (lambda (k.v)
+                                       (let ((key (car k.v))
+                                             (val (cdr k.v)))
+                                         (cons key (org-link-decode val)))) alist)))
+    (unless (assoc 'ref decoded-alist)
+      (error "No ref key provided."))
+    (when-let ((title (cdr (assoc 'title decoded-alist))))
+      (push (cons 'slug (org-roam--title-to-slug title)) decoded-alist))
+    (let* ((org-roam-capture-templates org-roam-ref-capture-templates)
+           (org-roam--capture-context 'ref)
+           (org-roam--capture-info decoded-alist)
+           (template (cdr (assoc 'template decoded-alist))))
+      (raise-frame)
+      (org-roam-capture nil template)
+      (message "Item captured.")))
+  nil)
+
+(defun org-roam-protocol-open-file (info)
+  "This handler simply opens the file with emacsclient.
+
+INFO is an alist containing additional information passed by the protocol URL.
+It should contain the FILE key, pointing to the path of the file to open.
+
+  Example protocol string:
+
+org-protocol://roam-file?file=/path/to/file.org"
+  (when-let ((file (plist-get info :file)))
+    (raise-frame)
+    (find-file file))
+  nil)
+
+(push '("org-roam-ref"  :protocol "roam-ref"   :function org-roam-protocol-open-ref)
+      org-protocol-protocol-alist)
+(push '("org-roam-file"  :protocol "roam-file"   :function org-roam-protocol-open-file)
+      org-protocol-protocol-alist)
+
+(provide 'org-roam-protocol)
+
+;;; org-roam-protocol.el ends here

org-roam-utils.el

@@ -1,169 +0,0 @@
-;;; org-roam-utils.el --- Roam Research replica with Org-mode -*- coding: utf-8; lexical-binding: t -*-
-
-;; Copyright © 2020 Jethro Kuan <jethrokuan95@gmail.com>
-
-;; Author: Jethro Kuan <jethrokuan95@gmail.com>
-;; URL: https://github.com/jethrokuan/org-roam
-;; Keywords: org-mode, roam, convenience
-;; Version: 0.1.2
-;; Package-Requires: ((emacs "26.1"))
-
-;; This file is NOT part of GNU Emacs.
-
-;; This program is free software; you can redistribute it and/or modify
-;; it under the terms of the GNU General Public License as published by
-;; the Free Software Foundation; either version 3, or (at your option)
-;; any later version.
-;;
-;; This program is distributed in the hope that it will be useful,
-;; but WITHOUT ANY WARRANTY; without even the implied warranty of
-;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-;; GNU General Public License for more details.
-;;
-;; You should have received a copy of the GNU General Public License
-;; along with GNU Emacs; see the file COPYING.  If not, write to the
-;; Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor,
-;; Boston, MA 02110-1301, USA.
-
-;;; Commentary:
-;;
-;; This library is an attempt at injecting Roam functionality into Org-mode.
-;; This is achieved primarily through building caches for forward links,
-;; backward links, and file titles.
-;;
-;;; Code:
-
-(require 'org)
-(require 'org-element)
-(require 'subr-x)
-(require 'cl-lib)
-
-(defun org-roam--file-name-extension (filename)
-  "Return file name extension for FILENAME.
-
-Like file-name-extension, but does not strip version number."
-  (save-match-data
-    (let ((file (file-name-nondirectory filename)))
-      (if (and (string-match "\\.[^.]*\\'" file)
-                     (not (eq 0 (match-beginning 0))))
-          (substring file (+ (match-beginning 0) 1))))))
-
-(defun org-roam--org-file-p (path)
-  "Check if PATH is pointing to an org file."
-  (let ((ext (org-roam--file-name-extension path)))
-    (or (string= ext "org")
-        (and
-         (string= ext "gpg")
-         (string= (org-roam--file-name-extension (file-name-sans-extension path)) "org")))))
-
-(defun org-roam--find-files (dir)
-  "Return all `org-roam' files in `DIR'."
-  (if (file-exists-p dir)
-      (let ((files (directory-files dir t "." t))
-            (dir-ignore-regexp (concat "\\(?:"
-                                       "\\."
-                                       "\\|\\.\\."
-                                       "\\)$"))
-            result)
-        (dolist (file files)
-          (cond
-           ((file-directory-p file)
-            (when (not (string-match dir-ignore-regexp file))
-              (setq result (append (org-roam--find-files file) result))))
-           ((and (file-readable-p file)
-                 (org-roam--org-file-p file))
-            (setq result (cons (file-truename file) result)))))
-        result)))
-
-(defun org-roam--parse-content (&optional file-path)
-  "Parse the current buffer, and return a list of items for processing."
-  (org-element-map (org-element-parse-buffer) 'link
-    (lambda (link)
-      (let ((type (org-element-property :type link))
-            (path (org-element-property :path link))
-            (start (org-element-property :begin link)))
-        (when (and (string= type "file")
-                   (org-roam--org-file-p path))
-          (goto-char start)
-          (let* ((element (org-element-at-point))
-                 (begin (or (org-element-property :content-begin element)
-                            (org-element-property :begin element)))
-                 (content (or (org-element-property :raw-value element)
-                              (buffer-substring
-                               begin
-                               (or (org-element-property :content-end element)
-                                   (org-element-property :end element)))))
-                 (content (string-trim content))
-                 (file-path (or file-path
-                                (file-truename (buffer-file-name (current-buffer))))))
-            (list :from file-path
-                  :to (file-truename (expand-file-name path (file-name-directory file-path)))
-                  :properties (list :content content :point begin))))))))
-
-(cl-defun org-roam--insert-item (item &key forward backward)
-  "Insert ITEM into FORWARD and BACKWARD cache.
-
-ITEM is of the form: (:from from-path :to to-path :properties (:content preview-content :point point))."
-  (pcase-let ((`(:from ,p-from :to ,p-to :properties ,props) item))
-    ;; Build forward-links
-    (let ((links (gethash p-from forward)))
-      (if links
-          (puthash p-from
-                   (if (member p-to links)
-                       links
-                     (cons p-to links)) forward)
-        (puthash p-from (list p-to) forward)))
-    ;; Build backward-links
-    (let ((contents-hash (gethash p-to backward)))
-      (if contents-hash
-          (if-let ((contents-list (gethash p-from contents-hash)))
-              (let ((updated (cons props contents-list)))
-                (puthash p-from updated contents-hash)
-                (puthash p-to contents-hash backward))
-            (progn
-              (puthash p-from (list props) contents-hash)
-              (puthash p-to contents-hash backward)))
-        (let ((contents-hash (make-hash-table :test #'equal)))
-          (puthash p-from (list props) contents-hash)
-          (puthash p-to contents-hash backward))))))
-
-(defun org-roam--extract-title ()
-  "Extract the title from `BUFFER'."
-  (org-element-map
-      (org-element-parse-buffer)
-      'keyword
-    (lambda (kw)
-      (when (string= (org-element-property :key kw) "TITLE")
-        (org-element-property :value kw)))
-    :first-match t))
-
-(defun org-roam--build-cache (dir)
-  "Build the org-roam caches in DIR."
-  (let ((backward-links (make-hash-table :test #'equal))
-        (forward-links (make-hash-table :test #'equal))
-        (file-titles (make-hash-table :test #'equal)))
-    (let* ((org-roam-files (org-roam--find-files dir))
-           (file-items (mapcar (lambda (file)
-                                 (with-temp-buffer
-                                   (insert-file-contents file)
-                                   (org-roam--parse-content file))) org-roam-files)))
-      (dolist (items file-items)
-        (dolist (item items)
-          (org-roam--insert-item
-           item
-           :forward forward-links
-           :backward backward-links)))
-      (dolist (file org-roam-files)
-        (with-temp-buffer
-          (insert-file-contents file)
-          (when-let ((title (org-roam--extract-title)))
-            (puthash file title file-titles)))
-        org-roam-files))
-    (list
-     :forward forward-links
-     :backward backward-links
-     :titles file-titles)))
-
-(provide 'org-roam-utils)
-
-;;; org-roam-utils.el ends here

tests/roam-files/alias.org

@@ -0,0 +1,2 @@
+#+ROAM_ALIAS: "a1" "a 2"
+#+TITLE: t1

tests/roam-files/bar.org

@@ -0,0 +1,3 @@
+#+TITLE: Bar
+
+This is file bar. Bar links to [[file:nested/bar.org][Nested Bar]].

tests/roam-files/f1.org

@@ -1,7 +0,0 @@
-#+TITLE: File 1
-
-link to [[file:nested/f1.org][Nested File 1]] 
-link to [[file:f2.org][File 2]]
-
-Arbitrary [[https://google.com][HTML]] link
-Arbitrary text

tests/roam-files/f3.org

@@ -1,5 +0,0 @@
-#+TITLE: File 3
-
-This file has a link to an file with no title.
-
-[[file:no-title.org][no-title]]

tests/roam-files/foo.org

@@ -0,0 +1,8 @@
+#+TITLE: Foo
+
+This is the foo file. It contains a link to [[file:bar.org][Bar]].
+
+To make the tests more robust, here are some arbitrary links:
+
+- [[https:google.com][Google]]
+- [[mailto:foo@john.com][mail to foo]]

tests/roam-files/nested/bar.org

@@ -0,0 +1,3 @@
+#+TITLE: Nested Bar
+
+This file is nested, 1 level deeper. It links to both [[file:../foo.org][Foo]] and [[file:foo.org][Nested Foo]].

tests/roam-files/nested/f1.org

@@ -1,4 +0,0 @@
-#+TITLE: Nested File 1
-
-Link to [[file:f2.org][Nested File 2]]
-Link to [[file:../f1.org][File 1]]

tests/roam-files/nested/f2.org

@@ -1,3 +0,0 @@
-#+TITLE: Nested File 2
-
-Link to [[file:f1.org][Nested File 1]]

tests/roam-files/nested/foo.org

@@ -1,3 +1,3 @@
-#+TITLE: File 2
+#+TITLE: Nested Foo
 
 This file has no links.

tests/roam-files/no-title.org

@@ -1 +1,3 @@
 no title in this file :O
+
+links to itself, with no title: [[file:no-title.org][no-title]]

tests/roam-files/unlinked.org

@@ -0,0 +1,3 @@
+#+TITLE: Unlinked
+
+Nothing links here :(

tests/roam-files/web_ref.org

@@ -0,0 +1 @@
+#+ROAM_KEY: https://google.com/

tests/test-org-roam.el

@@ -43,188 +43,261 @@
   "Directory containing org-roam test org files.")
 
 (defun org-roam--test-init ()
+  (org-roam--db-close)
   (let ((original-dir org-roam--tests-directory)
         (new-dir (expand-file-name (make-temp-name "org-roam") temporary-file-directory)))
     (copy-directory original-dir new-dir)
-    (setq org-roam-directory new-dir))
-  (org-roam-mode +1))
-
-(defun org-roam--test-build-cache ()
-  "Builds the caches synchronously."
-  (let ((cache (org-roam--build-cache org-roam-directory)))
-    (setq org-roam-forward-links-cache (plist-get cache :forward))
-    (setq org-roam-backward-links-cache (plist-get cache :backward))
-    (setq org-roam-titles-cache (plist-get cache :titles))
-    (setq org-roam-cache-initialized t)))
+    (setq org-roam-directory new-dir)
+    (org-roam-mode +1)))
 
 ;;; Tests
-(describe "org-roam--build-cache-async"
+(describe "org-roam-build-cache"
           (it "initializes correctly"
               (org-roam--test-init)
-              (expect org-roam-cache-initialized :to-be nil)
-              (expect (hash-table-count org-roam-forward-links-cache) :to-be 0)
-              (expect (hash-table-count org-roam-backward-links-cache) :to-be 0)
-              (expect (hash-table-count org-roam-titles-cache) :to-be 0)
+              (org-roam-build-cache)
 
-              (org-roam--build-cache-async)
-              (sleep-for 3) ;; Because it's async
+              ;; Cache
+              (expect (caar (org-roam-sql [:select (funcall count) :from files])) :to-be 8)
+              (expect (caar (org-roam-sql [:select (funcall count) :from file-links])) :to-be 5)
+              (expect (caar (org-roam-sql [:select (funcall count) :from titles])) :to-be 8)
+              (expect (caar (org-roam-sql [:select (funcall count) :from titles
+                                                   :where titles :is-null])) :to-be 2)
+              (expect (caar (org-roam-sql [:select (funcall count) :from refs])) :to-be 1)
 
-              ;; Caches should be populated
-              (expect org-roam-cache-initialized :to-be t)
-              (expect (hash-table-count org-roam-forward-links-cache) :to-be 4)
-              (expect (hash-table-count org-roam-backward-links-cache) :to-be 5)
-              (expect (hash-table-count org-roam-titles-cache) :to-be 5)
+              ;; TODO Test files
 
-              ;; Forward cache
-              (let ((f1 (gethash (abs-path "f1.org") org-roam-forward-links-cache))
-                    (f2 (gethash (abs-path "f2.org") org-roam-forward-links-cache))
-                    (nested-f1 (gethash (abs-path "nested/f1.org") org-roam-forward-links-cache))
-                    (nested-f2 (gethash (abs-path "nested/f2.org") org-roam-forward-links-cache))
-                    (expected-f1 (list (abs-path "nested/f1.org")
-                                       (abs-path "f2.org")))
-                    (expected-nested-f1 (list (abs-path "nested/f2.org")
-                                              (abs-path "f1.org")))
-                    (expected-nested-f2 (list (abs-path "nested/f1.org"))))
+              ;; Links -- File-from
+              (expect (caar (org-roam-sql [:select (funcall count) :from file-links
+                                                   :where (= file-from $s1)]
+                                          (abs-path "foo.org"))) :to-be 1)
+              (expect (caar (org-roam-sql [:select (funcall count) :from file-links
+                                                   :where (= file-from $s1)]
+                                          (abs-path "nested/bar.org"))) :to-be 2)
 
-                (expect f1 :to-have-same-items-as expected-f1)
-                (expect f2 :to-be nil)
-                (expect nested-f1 :to-have-same-items-as expected-nested-f1)
-                (expect nested-f2 :to-have-same-items-as expected-nested-f2))
+              ;; Links -- File-to
+              (expect (caar (org-roam-sql [:select (funcall count) :from file-links
+                                                   :where (= file-to $s1)]
+                                          (abs-path "nested/foo.org"))) :to-be 1)
+              (expect (caar (org-roam-sql [:select (funcall count) :from file-links
+                                                   :where (= file-to $s1)]
+                                          (abs-path "nested/bar.org"))) :to-be 1)
+              (expect (caar (org-roam-sql [:select (funcall count) :from file-links
+                                                   :where (= file-to $s1)]
+                                          (abs-path "unlinked.org"))) :to-be 0)
+              ;; TODO Test titles
+              (expect (org-roam-sql [:select * :from titles])
+                      :to-have-same-items-as
+                      (list (list (abs-path "alias.org")
+                                  (list "t1" "a1" "a 2"))
+                            (list (abs-path "bar.org")
+                                  (list "Bar"))
+                            (list (abs-path "foo.org")
+                                  (list "Foo"))
+                            (list (abs-path "nested/bar.org")
+                                  (list "Nested Bar"))
+                            (list (abs-path "nested/foo.org")
+                                  (list "Nested Foo"))
+                            (list (abs-path "no-title.org") nil)
+                            (list (abs-path "web_ref.org") nil)
+                            (list (abs-path "unlinked.org")
+                                  (list "Unlinked"))))
 
-              ;; Backward cache
-              (let ((f1 (hash-table-keys (gethash (abs-path "f1.org") org-roam-backward-links-cache)))
-                    (f2 (hash-table-keys (gethash (abs-path "f2.org") org-roam-backward-links-cache)))
-                    (nested-f1 (hash-table-keys(gethash (abs-path "nested/f1.org") org-roam-backward-links-cache)))
-                    (nested-f2 (hash-table-keys (gethash (abs-path "nested/f2.org") org-roam-backward-links-cache)))
-                    (expected-f1 (list (abs-path "nested/f1.org")))
-                    (expected-f2 (list (abs-path "f1.org")))
-                    (expected-nested-f1 (list (abs-path "nested/f2.org")
-                                              (abs-path "f1.org")))
-                    (expected-nested-f2 (list (abs-path "nested/f1.org"))))
-                (expect f1 :to-have-same-items-as expected-f1)
-                (expect f2 :to-have-same-items-as expected-f2)
-                (expect nested-f1 :to-have-same-items-as expected-nested-f1)
-                (expect nested-f2 :to-have-same-items-as expected-nested-f2))
+              (expect (org-roam-sql [:select * :from refs])
+                      :to-have-same-items-as
+                      (list (list "https://google.com/" (abs-path "web_ref.org"))))
 
-              ;; Titles Cache
-              (expect (gethash (abs-path "f1.org") org-roam-titles-cache) :to-equal "File 1")
-              (expect (gethash (abs-path "f2.org") org-roam-titles-cache) :to-equal "File 2")
-              (expect (gethash (abs-path "nested/f1.org") org-roam-titles-cache) :to-equal "Nested File 1")
-              (expect (gethash (abs-path "nested/f2.org") org-roam-titles-cache) :to-equal "Nested File 2")
-              (expect (gethash (abs-path "no-title.org") org-roam-titles-cache) :to-be nil)))
+              ;; Expect rebuilds to be really quick (nothing changed)
+              (expect (org-roam-build-cache)
+                      :to-equal
+                      (list :files 0 :links 0 :titles 0 :refs 0 :deleted 0))))
 
 (describe "org-roam-insert"
           (before-each
            (org-roam--test-init)
-           (org-roam--clear-cache)
-           (org-roam--test-build-cache))
+           (org-roam--db-clear)
+           (org-roam-build-cache))
 
-          (it "temp1 -> f1"
+          (it "temp1 -> foo"
               (let ((buf (org-roam--test-find-new-file "temp1.org")))
                 (with-current-buffer buf
                   (with-simulated-input
-                   "File SPC 1 RET"
+                   "Foo RET"
                    (org-roam-insert nil))))
-              (expect (buffer-string) :to-match (regexp-quote "file:f1.org")))
+              (expect (buffer-string) :to-match (regexp-quote "file:foo.org")))
 
-          (it "temp2 -> nested/f1"
+          (it "temp2 -> nested/foo"
               (let ((buf (org-roam--test-find-new-file "temp2.org")))
                 (with-current-buffer buf
                   (with-simulated-input
-                   "Nested SPC File SPC 1 RET"
+                   "Nested SPC Foo RET"
                    (org-roam-insert nil))))
-              (expect (buffer-string) :to-match (regexp-quote "file:nested/f1.org")))
+              (expect (buffer-string) :to-match (regexp-quote "file:nested/foo.org")))
 
-          (it "nested/temp3 -> f1"
+          (it "nested/temp3 -> foo"
               (let ((buf (org-roam--test-find-new-file "nested/temp3.org")))
                 (with-current-buffer buf
                   (with-simulated-input
-                   "File SPC 1 RET"
+                   "Foo RET"
                    (org-roam-insert nil))))
-              (expect (buffer-string) :to-match (regexp-quote "file:../f1.org")))
+              (expect (buffer-string) :to-match (regexp-quote "file:../foo.org")))
 
-          (it "a/b/temp4 -> nested/f1"
+          (it "a/b/temp4 -> nested/foo"
               (let ((buf (org-roam--test-find-new-file "a/b/temp4.org")))
                 (with-current-buffer buf
                   (with-simulated-input
-                   "Nested SPC File SPC 1 RET"
+                   "Nested SPC Foo RET"
                    (org-roam-insert nil))))
-              (expect (buffer-string) :to-match (regexp-quote "file:../../nested/f1.org"))))
+              (expect (buffer-string) :to-match (regexp-quote "file:../../nested/foo.org"))))
 
 (describe "rename file updates cache"
           (before-each
            (org-roam--test-init)
-           (org-roam--clear-cache)
-           (org-roam--test-build-cache))
+           (org-roam--db-clear)
+           (org-roam-build-cache))
 
-          (it "f1 -> new_f1"
-              (rename-file (abs-path "f1.org")
-                           (abs-path "new_f1.org"))
+          (it "foo -> new_foo"
+              (rename-file (abs-path "foo.org")
+                           (abs-path "new_foo.org"))
               ;; Cache should be cleared of old file
-              (expect (gethash (abs-path "f1.org")  org-roam-forward-links-cache) :to-be nil)
-              (expect (->> org-roam-backward-links-cache
-                           (gethash (abs-path "nested/f1.org"))
-                           (hash-table-keys)
-                           (member (abs-path "f1.org"))) :to-be nil)
+              (expect (caar (org-roam-sql [:select (funcall count)
+                                           :from titles
+                                           :where (= file $s1)]
+                                          (abs-path "foo.org"))) :to-be 0)
+              (expect (caar (org-roam-sql [:select (funcall count)
+                                           :from refs
+                                           :where (= file $s1)]
+                                          (abs-path "foo.org"))) :to-be 0)
+              (expect (caar (org-roam-sql [:select (funcall count)
+                                           :from file-links
+                                           :where (= file-from $s1)]
+                                          (abs-path "foo.org"))) :to-be 0)
 
-              (expect (->> org-roam-forward-links-cache
-                           (gethash (abs-path "new_f1.org"))) :not :to-be nil)
+              ;; Cache should be updated
+              (expect (org-roam-sql [:select [file-to]
+                                     :from file-links
+                                     :where (= file-from $s1)]
+                                    (abs-path "new_foo.org"))
+                      :to-have-same-items-as
+                      (list (list (abs-path "bar.org"))))
+              (expect (org-roam-sql [:select [file-from]
+                                     :from file-links
+                                     :where (= file-to $s1)]
+                                    (abs-path "new_foo.org"))
+                      :to-have-same-items-as
+                      (list (list (abs-path "nested/bar.org"))))
 
-              (expect (->> org-roam-forward-links-cache
-                           (gethash (abs-path "new_f1.org"))
-                           (member (abs-path "nested/f1.org"))) :not :to-be nil)
               ;; Links are updated
               (expect (with-temp-buffer
-                        (insert-file-contents (abs-path "nested/f1.org"))
-                        (buffer-string)) :to-match (regexp-quote "[[file:../new_f1.org][File 1]]")))
+                        (insert-file-contents (abs-path "nested/bar.org"))
+                        (buffer-string))
+                      :to-match
+                      (regexp-quote "[[file:../new_foo.org][Foo]]")))
 
-
-          (it "f1 -> f1 with spaces"
-              (rename-file (abs-path "f1.org")
-                           (abs-path "f1 with spaces.org"))
+          (it "foo -> foo with spaces"
+              (rename-file (abs-path "foo.org")
+                           (abs-path "foo with spaces.org"))
               ;; Cache should be cleared of old file
-              (expect (gethash (abs-path "f1.org")  org-roam-forward-links-cache) :to-be nil)
-              (expect (->> org-roam-backward-links-cache
-                           (gethash (abs-path "nested/f1.org"))
-                           (hash-table-keys)
-                           (member (abs-path "f1.org"))) :to-be nil)
+              (expect (caar (org-roam-sql [:select (funcall count)
+                                           :from titles
+                                           :where (= file $s1)]
+                                          (abs-path "foo.org"))) :to-be 0)
+              (expect (caar (org-roam-sql [:select (funcall count)
+                                           :from refs
+                                           :where (= file $s1)]
+                                          (abs-path "foo.org"))) :to-be 0)
+              (expect (caar (org-roam-sql [:select (funcall count)
+                                           :from file-links
+                                           :where (= file-from $s1)]
+                                          (abs-path "foo.org"))) :to-be 0)
+
+              ;; Cache should be updated
+              (expect (org-roam-sql [:select [file-to]
+                                     :from file-links
+                                     :where (= file-from $s1)]
+                                    (abs-path "foo with spaces.org"))
+                      :to-have-same-items-as
+                      (list (list (abs-path "bar.org"))))
+              (expect (org-roam-sql [:select [file-from]
+                                     :from file-links
+                                     :where (= file-to $s1)]
+                                    (abs-path "foo with spaces.org"))
+                      :to-have-same-items-as
+                      (list (list (abs-path "nested/bar.org"))))
+
               ;; Links are updated
               (expect (with-temp-buffer
-                        (insert-file-contents (abs-path "nested/f1.org"))
-                        (buffer-string)) :to-match (regexp-quote "[[file:../f1 with spaces.org][File 1]]")))
+                        (insert-file-contents (abs-path "nested/bar.org"))
+                        (buffer-string))
+                      :to-match
+                      (regexp-quote "[[file:../foo with spaces.org][Foo]]")))
 
           (it "no-title -> meaningful-title"
               (rename-file (abs-path "no-title.org")
                            (abs-path "meaningful-title.org"))
               ;; File has no forward links
-              (expect (gethash (abs-path "no-title.org")  org-roam-forward-links-cache) :to-be nil)
-              (expect (gethash (abs-path "meaningful-title.org")  org-roam-forward-links-cache) :to-be nil)
-
-              (expect (->> org-roam-forward-links-cache
-                           (gethash (abs-path "f3.org"))
-                           (member (abs-path "no-title.org"))) :to-be nil)
-
-              (expect (->> org-roam-forward-links-cache
-                           (gethash (abs-path "f3.org"))
-                           (member (abs-path "meaningful-title.org"))) :not :to-be nil)
+              (expect (caar (org-roam-sql [:select (funcall count)
+                                           :from file-links
+                                           :where (= file-from $s1)]
+                                          (abs-path "no-title.org"))) :to-be 0)
+              (expect (caar (org-roam-sql [:select (funcall count)
+                                           :from file-links
+                                           :where (= file-from $s1)]
+                                          (abs-path "meaningful-title.org"))) :to-be 1)
 
               ;; Links are updated with the appropriate name
               (expect (with-temp-buffer
-                        (insert-file-contents (abs-path "f3.org"))
-                        (buffer-string)) :to-match (regexp-quote "[[file:meaningful-title.org][meaningful-title]]"))))
+                        (insert-file-contents (abs-path "meaningful-title.org"))
+                        (buffer-string))
+                      :to-match
+                      (regexp-quote "[[file:meaningful-title.org][meaningful-title]]")))
+
+          (it "web_ref -> hello"
+              (expect (org-roam-sql
+                       [:select [file] :from refs
+                        :where (= ref $s1)]
+                       "https://google.com/")
+                      :to-equal
+                      (list (list (abs-path "web_ref.org"))))
+              (rename-file (abs-path "web_ref.org")
+                           (abs-path "hello.org"))
+              (expect (org-roam-sql
+                       [:select [file] :from refs
+                        :where (= ref $s1)]
+                       "https://google.com/")
+                      :to-equal (list (list (abs-path "hello.org"))))
+              (expect (caar (org-roam-sql
+                             [:select [ref] :from refs
+                              :where (= file $s1)]
+                             (abs-path "web_ref.org")))
+                      :to-equal nil)))
 
 (describe "delete file updates cache"
           (before-each
            (org-roam--test-init)
-           (org-roam--clear-cache)
-           (org-roam--test-build-cache))
-          (it "delete f1"
-              (delete-file (abs-path "f1.org"))
-              (expect (->> org-roam-forward-links-cache
-                           (gethash (abs-path "f1.org"))) :to-be nil)
-              (expect (->> org-roam-backward-links-cache
-                           (gethash (abs-path "nested/f1.org"))
-                           (gethash (abs-path "f1.org"))) :to-be nil)
-              (expect (->> org-roam-backward-links-cache
-                           (gethash (abs-path "nested/f1.org"))
-                           (gethash (abs-path "nested/f2.org"))) :not :to-be nil)))
+           (org-roam--db-clear)
+           (org-roam-build-cache)
+           (sleep-for 1))
+
+          (it "delete foo"
+              (delete-file (abs-path "foo.org"))
+              (expect (caar (org-roam-sql [:select (funcall count)
+                                           :from titles
+                                           :where (= file $s1)]
+                                          (abs-path "foo.org"))) :to-be 0)
+              (expect (caar (org-roam-sql [:select (funcall count)
+                                           :from refs
+                                           :where (= file $s1)]
+                                          (abs-path "foo.org"))) :to-be 0)
+              (expect (caar (org-roam-sql [:select (funcall count)
+                                           :from file-links
+                                           :where (= file-from $s1)]
+                                          (abs-path "foo.org"))) :to-be 0))
+
+          (it "delete web_ref"
+              (expect (org-roam-sql [:select * :from refs])
+                      :to-have-same-items-as
+                      (list (list "https://google.com/" (abs-path "web_ref.org"))))
+              (delete-file (abs-path "web_ref.org"))
+              (expect (org-roam-sql [:select * :from refs])
+                      :to-have-same-items-as
+                      (list))))