(feat): Implement org-roam-flow

When using gpg encrypted files it might happen (intended) that on a
certain machine there is no key to decrypt that file.  Currently an
error of type 'file-error' will be raised and the cache building
process will be aborted.  So in a certain sense org-roam will not
be functional in that case.  Furthermore, when the the cache building
process is run from the emacs initialisation it will come to a halt as
well and the user will be left with a halfworking emacs instance.

This patch changes the behaviour to just skipping over offending files
while removing them from the db at the same time.  Here an offending
file is any file that cannot be read for indexing.  As it stands this
case only works reliably for the case of missing gpg keys.

An error buffer will still show up and be prominently displayed,
besides a log to the message buffer.

Implementation note: This io error will only be caught during the
first part of the cache rebuilding ("files and headlines").  If such
an error would occur during the second part ("rebuild the rest")
another (more severe) cause might be the problem and the user better
be informed the hard way (i.e. the same behaviour as before).

.dir-locals.el

@@ -1,4 +1,6 @@
 ;;; Directory Local Variables
 ;;; For more information see (info "(emacs) Directory Variables")
 
-((emacs-lisp-mode . ((eval . (require 'org-roam-dev)))))
+((emacs-lisp-mode
+  (eval . (require 'org-roam-dev))
+  (eval . (org-roam-dev-mode))))

.github/workflows/stale.yml

@@ -1,19 +0,0 @@
-name: Mark stale issues and pull requests
-
-on:
-  schedule:
-  - cron: "0 0 * * *"
-
-jobs:
-  stale:
-
-    runs-on: ubuntu-latest
-
-    steps:
-    - uses: actions/stale@v1
-      with:
-        repo-token: ${{ secrets.GITHUB_TOKEN }}
-        stale-issue-message: 'Stale issue message'
-        stale-pr-message: 'Stale pull request message'
-        stale-issue-label: 'no-issue-activity'
-        stale-pr-label: 'no-pr-activity'

.github/workflows/test.yml

@@ -54,9 +54,11 @@ jobs:
     # The "all" rule is not used, because it treats compilation warnings
     # as failures, so linting and testing are run as separate steps.
 
+    # org-roam-compat is excluded from linting because it contains
+    # symbols/aliases from other packages
     - name: Lint
       continue-on-error: false
-      run: ./makem.sh -vv --sandbox $SANDBOX_DIR lint
+      run: ./makem.sh -vv --sandbox $SANDBOX_DIR --exclude org-roam-compat.el lint
 
     - name: Test
       if: always()  # Run test even if linting fails.

BACKERS.md

@@ -0,0 +1,8 @@
+# Backers
+
+Many thanks to the following backers, your contributions are greatly appreciated!
+
+- Nathan Tran
+- Burke Libbey
+- forkrul
+- Andreas Stuhlmüller

CHANGELOG.md

@@ -1,11 +1,80 @@
 # Changelog
 
+## 1.2.2 (TBD)
+
+### Breaking Changes
+
+### Features
+
+- [#974](https://github.com/org-roam/org-roam/pull/974) Protect region targeted by `org-roam-insert`
+- [#994](https://github.com/org-roam/org-roam/pull/994) Simplify org-roam-store-link
+
+### Bugfixes
+
+## 1.2.1 (27-07-2020)
+
+This release consisted of a big deal of refactoring and bug fixes. Notably, we fixed several catastrophic failures on db builds with bad setups (#854), and modularized tag and title extractions.
+
+We also added some new features that had been a long time coming:
+
+1. We made the backlinks more outline-friendly by also showing the outline hierarchy for a backlink (#863)
+2. We now support nested captures, which is crucial for `org-roam-protocol` (#966)
+
+### Breaking Changes
+- [#908](https://github.com/org-roam/org-roam/pull/908) Normalized titles in database. May break external packages that rely on unnormalized titles.
+
+### Features
+- [#814](https://github.com/org-roam/org-roam/pull/814) Implement `org-roam-insert-immediate`
+- [#833](https://github.com/org-roam/org-roam/pull/833) Add customization of file titles with `org-roam-title-to-slug-function`. 
+- [#839](https://github.com/org-roam/org-roam/pull/839) Return selected file from `org-roam-insert`
+- [#847](https://github.com/org-roam/org-roam/pull/847) Add GC threshold `org-roam-db-gc-threshold` to temporarily change the threshold on expensive operations.
+- [#847](https://github.com/org-roam/org-roam/pull/847) Use sqlite3 transactions instead of storing the values to be inserted.
+- [#851](https://github.com/org-roam/org-roam/pull/851) Add `'first-directory'` option for `org-roam-tag-sources`
+- [#863](https://github.com/org-roam/org-roam/pull/863) Display outline hierarchy in backlinks buffer
+- [#898](https://github.com/org-roam/org-roam/pull/898) Add `org-roam-random-note` to browse a random note.
+- [#900](https://github.com/org-roam/org-roam/pull/900) Support and index all valid org links
+- [#966](https://github.com/org-roam/org-roam/pull/966) Enable nested captures
+
+### Bugfixes
+
+- [#854](https://github.com/org-roam/org-roam/pull/854) Warn instead of fail when duplicate refs and IDs exist.
+- [#857](https://github.com/org-roam/org-roam/pull/857) Fix tag extraction for symlinked directories.
+- [#894](https://github.com/org-roam/org-roam/pull/894) Perform link fixes on all Org-roam files
+- [#952](https://github.com/org-roam/org-roam/pull/952) Cache `${foo}` template variables so they do not need to be re-entered twice
+
+## 1.2.0 (12-06-2020)
+
+In this release, we improved the linking process by achieving feature parity between links to files and links to headlines. Before, we used the `file:foo::*bar` format to link to the headline `bar` in file `foo`, but this was prone to breakage upon renaming the file or modifying the headline. This is not the case anymore. Now, we use `org-id` to create IDs for those headlines, which are then stored in our database to compute the relationships and jump around. Note that this will work even if you’re not using `org-id` in your global configuration for Org-mode.
+
+This is a major step forward. Supporting the in-file structure of Org-mode files means that we can interface with many of its core-features like TODOs, properties, priorities, etc. UX will have to be figured out, but this release ushers in a new age in terms of functionalities.
+
+We also add `org-roam-unlinked-references`, which naively finds text that could be references to the current Org-roam file.
+
+### Breaking Changes
+
+- [#701](https://github.com/org-roam/org-roam/pull/701) Use `emacsql-sqlite3` instead of `emacsql-sqlite` for better Windows compatibility. This requires the presence of the standard `sqlite3` binary on your machine.
+- [#750](https://github.com/org-roam/org-roam/pull/750) Deprecate `org-roam-buffer-no-delete-other-windows` in favour of `org-roam-buffer-window-parameters`.
+
+### Features
+
+- [#787](https://github.com/org-roam/org-roam/pull/787) Add `org-roam-unlinked-references`
+- [#783](https://github.com/org-roam/org-roam/pull/783) Add support for headlines
+- [#757](https://github.com/org-roam/org-roam/pull/757) Roam global properties are now case-insensitive
+- [#680](https://github.com/org-roam/org-roam/pull/680) , [#703](https://github.com/org-roam/org-roam/pull/703), [#708](https://github.com/org-roam/org-roam/pull/708) Add `org-roam-doctor` checkers for `ROAM_*` properties
+- [#664](https://github.com/org-roam/org-roam/pull/664) Add support for shelling out to `rg` and `find` in `org-roam--list-files`
+- [#679](https://github.com/org-roam/org-roam/pull/679), [#683](https://github.com/org-roam/org-roam/pull/683) Building of the graph now happens in a separate process
+
+### Bugfixes
+- [#714](https://github.com/org-roam/org-roam/pull/714) No longer print citelinks in backlinks buffer if there is no `ROAM_KEY` property or `org-ref` is not installed
+- [#759](https://github.com/org-roam/org-roam/pull/759), [#760](https://github.com/org-roam/org-roam/pull/760) Tags are only added to the tags table if there are actually tags present
+- [#700](https://github.com/org-roam/org-roam/pull/700), [#733](https://github.com/org-roam/org-roam/pull/733) Allow symlinks within the `org-roam` directory
+
 ## 1.1.1 (18-05-2020)
 
 In this release, we added two new features:
 
 1. `org-roam-doctor`: a linting system that helps you discover possible problems with your Org-roam files. This is in the spirit of keeping notes in top quality.
-2. A tagging system: one can now use sub-directories, and the `#+ROAM_TAG` key add additional meta data to notes. For more information, see [here](https://org-roam.github.io/org-roam/manual/Tags.html#Tags).
+2. A tagging system: one can now use sub-directories, and the `#+roam_tags` key add additional meta data to notes. For more information, see [here](https://www.orgroam.com/manual/Tags.html#Tags).
 
 As usual, this release comes with a multitude of bug-fixes and refactorings.
 

README.md

@@ -4,6 +4,11 @@
 
 ## Synopsis
 
+> **NOTE:** Org-roam builds upon Emacs and Org-mode, both of which are intricate
+> tools that require time investment for mastery. This makes Org-roam less
+> friendly for beginners, but extremely powerful for those familiar with the
+> ecosystem, or willing to invest effort in it.
+
 Org-roam is a [Roam][roamresearch] replica built on top of the
 all-powerful [Org-mode][org].
 
@@ -50,6 +55,7 @@ Here's a sample configuration with using `use-package`:
 
 ```emacs-lisp
 (use-package org-roam
+      :ensure t
       :hook
       (after-init . org-roam-mode)
       :custom
@@ -57,9 +63,10 @@ Here's a sample configuration with using `use-package`:
       :bind (:map org-roam-mode-map
               (("C-c n l" . org-roam)
                ("C-c n f" . org-roam-find-file)
-               ("C-c n g" . org-roam-show-graph))
+               ("C-c n g" . org-roam-graph-show))
               :map org-mode-map
-              (("C-c n i" . org-roam-insert))))
+              (("C-c n i" . org-roam-insert))
+              (("C-c n I" . org-roam-insert-immediate))))
 ```
 
 `org-roam-graph` by default expects to find the `dot` executable
@@ -72,6 +79,25 @@ For more detailed installation and configuration instructions (including for
 Doom and Spacemacs users), please see [the
 documentation][docs].
 
+## Frequently-asked Questions
+
+Q: How do I create a note whose title already matches one of the candidates (e.g. creating `bar` when `barricade` already exists)?
+
+A: With `ivy`, you need to press `C-M-j` to use the current input instead of the nearest candidate. (Source: [`ivy`’s
+  FAQ](https://github.com/abo-abo/swiper#frequently-asked-questions))
+
+## Getting Help
+
+Before creating a new topic/issue, please be mindful of our time and ensure
+that it has not already been addressed on
+[GitHub][issues] or on
+[Discourse][discourse].
+
+- If you are new to Emacs and have problem setting up Org-roam, please ask your question on [Slack, channel #how-do-i][slack].
+- For quick questions, please ask them on [Slack, channel #troubleshooting][slack].
+- If something is not working as it should, or if you would like to suggest a new feature, please [create a new issue][issues].
+- If you have questions about your workflow with the slip-box method, please find a relevant topic on [Discourse][discourse], or create a new one.
+
 ## Knowledge Bases using Org-roam
 
 - [Jethro Kuan](https://braindump.jethro.dev/)
@@ -95,6 +121,7 @@ General Public License, Version 3
 [roamresearch]: https://www.roamresearch.com/
 [org]: https://orgmode.org/
 [badge-license]: https://img.shields.io/badge/license-GPL_3-green.svg
-[docs]: https://org-roam.github.io/org-roam/manual/
+[docs]: https://www.orgroam.com/manual/
 [discourse]: https://org-roam.discourse.group/
 [slack]: https://join.slack.com/t/orgroam/shared_invite/zt-deoqamys-043YQ~s5Tay3iJ5QRI~Lxg
+[issues]: https://github.com/org-roam/org-roam/issues

doc/css/styles.css

@@ -0,0 +1,85 @@
+body {
+  margin: 20px;
+  background: #303030;
+  color: #f5f5f5;
+  font-size: 1.6rem;
+  font-weight: 300;
+}
+
+.container {
+  max-width: 800px;
+}
+
+.nav {
+  margin-top: 5rem;
+}
+
+.nav-item a {
+  /* color: #797979; */
+  color: #bebebe;
+  text-decoration: none;
+  font-family: "Consolas", "Monaco", "Menlo", monospace;
+}
+
+.nav-item a:hover {
+  cursor: pointer;
+  /* color: #4d4a4a; */
+  color: #dcdcdc;
+  border: none;
+}
+
+.hero {
+  color: #f5f5f5;
+}
+
+.hero div {
+  margin-bottom: 5rem;
+}
+
+.hero h3 {
+  /* font-family: "Edelsans", sans-serif; */
+  font-size: 5rem;
+  margin-bottom: 0;
+}
+
+.hero h5 {
+  margin-top: 1rem;
+  margin-bottom: 0;
+  /* font-weight: 200; */
+  font-weight: 250;
+}
+
+.grid-demo-col {
+  background: #e4e4e4;
+}
+
+a {
+  color: #cd86ff;
+}
+
+a:hover {
+  color: #dbb9f3;
+}
+
+img {
+  max-width: 100%;
+}
+
+.footer-links {
+  font-size: 1.5rem;
+  color: #f5f5f5;
+  text-decoration: none;
+  font-weight: 250;
+}
+
+.header {
+  font-weight: 200;
+}
+
+#footer {
+  margin-bottom: 3rem;
+}
+
+#footer h5 {
+  margin-bottom: 1.5rem;
+}

doc/index.html

@@ -1,3 +1,175 @@
-<html>
+<!DOCTYPE html>
-  <h1>--> <a href="manual/index.html">MANUAL HERE</a> <--</h1>
+<html lang="en">
+  <head>
+    <meta charset="utf-8" />
+    <meta http-equiv="X-UA-Compatible" content="IE=edge" />
+    <meta name="viewport" content="width=device-width, initial-scale=1" />
+    <meta name="description" content="An Effortless PKM System." />
+    <meta name="author" content="Org-roam Contributors" />
+
+    <title>Org-roam</title>
+
+    <link rel="stylesheet" href="https://unpkg.com/wingcss" />
+    <link rel="stylesheet" type="text/css" href="./css/styles.css" />
+
+    <script src="https://code.iconify.design/1/1.0.6/iconify.min.js"></script>
+  </head>
+
+  <body>
+    <section class="hero container center text-center">
+      <div>
+        <img src="./img/logo.png" alt="Org-roam Logo" height="200" />
+        <h3>Org-roam</h3>
+        <h5>A plain-text personal knowledge management system.</h5>
+      </div>
+    </section>
+
+    <section class="container center">
+      <div>
+        <img src="./img/screenshot.png" alt="screenshot" />
+      </div>
+    </section>
+
+    <section id="features" class="container center">
+      <div>
+        <h3 class="header">FEATURES</h3>
+        <div class="row">
+          <div class="col">
+            <h5 class="left">
+              <iconify-icon data-icon="fxemoji:lock"></iconify-icon> &nbsp;
+              Private and Secure
+            </h5>
+            <p class="content">
+              Edit your personal wiki completely offline, entirely in your
+              control. Encrypt your notes with GPG. Take lasting notes in
+              plain-text.
+            </p>
+          </div>
+          <div class="col">
+            <h5 class="left">
+              <iconify-icon data-icon="flat-color-icons:link"></iconify-icon>
+              &nbsp; Make Connections
+            </h5>
+            <p class="content">
+              Connect notes and thoughts together with ease using backlinks.
+              Discover surprising and previously unseen connections in your
+              notes with the built-in graph visualization.
+            </p>
+          </div>
+        </div>
+
+        <div class="row">
+          <div class="col">
+            <h5 class="left">
+              <iconify-icon data-icon="twemoji:puzzle-piece"></iconify-icon>
+              &nbsp; Extensible and Powerful
+            </h5>
+            <p class="content">
+              Leverage Emacs' fantastic text-editing interface, and the mature
+              Emacs and Org-mode ecosystem of packages.
+            </p>
+          </div>
+          <div class="col">
+            <h5 class="left">
+              <iconify-icon data-icon="logos:git-icon"></iconify-icon> &nbsp;
+              Free and Open Source
+            </h5>
+            <p class="content">
+              Org-roam is licensed under the GNU General Public License version
+              3 or later. You are free to extend its functionality and
+              contribute back. Find
+              us <a href="https://github.com/org-roam/org-roam">here</a>.
+            </p>
+          </div>
+        </div>
+      </div>
+    </section>
+    
+    <section id="links" class="container">
+      <div>
+        <h3 class="header">LINKS</h3> New to Emacs and Org-mode, and trying to
+        find your way around? Org-roam has an inclusive community of users
+        passionate about Personal Knowledge Management -- we're happy to help!
+        You can find us on Discourse and Slack.
+        
+        <ul>
+            <li>Read our documentation within Emacs, or on the <a href="https://www.orgroam.com/manual/">Online Manual</a></li>
+            <li>Participate in our forum discussions on <a href="https://org-roam.discourse.group">Discourse</a></li>
+            <li>Chat with us on <a href="https://join.slack.com/t/orgroam/shared_invite/zt-deoqamys-043YQ~s5Tay3iJ5QRI~Lxg">Slack</a></li>
+          </ul>
+      </div>
+    </section>
+
+    <section id="footer" class="row container">
+      <div class="col">
+        <div class="row">
+          <h5 class="header">Ecosystem</h5>
+        </div>
+
+        <div class="row">
+          <a
+            class="content footer-links"
+            href="https://github.com/org-roam/org-roam-bibtex"
+            >org-roam-bibtex</a
+          >
+        </div>
+        <div class="row">
+          <a
+            class="content footer-links"
+            href="https://github.com/org-roam/org-roam-server"
+            >org-roam-server</a
+          >
+        </div>
+        <div class="row">
+          <a
+            class="content footer-links"
+            href="https://github.com/org-roam/company-org-roam"
+            >company-org-roam</a
+          >
+        </div>
+      </div>
+
+      <div class="col">
+        <div class="row">
+          <h5 class="header">Resources</h5>
+        </div>
+
+        <div class="row">
+          <a
+            class="content footer-links"
+            href="https://github.com/org-roam/org-roam/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc"
+            >Open Issues</a
+          >
+        </div>
+        <div class="row">
+          <a
+            class="content footer-links"
+            href="https://github.com/org-roam/org-roam/releases"
+            >Releases</a
+          >
+        </div>
+      </div>
+
+      <div class="col">
+        <div class="row">
+          <h5 class="header">Connect</h5>
+        </div>
+
+        <div class="row">
+          <a
+            class="content footer-links"
+            href="https://join.slack.com/t/orgroam/shared_invite/zt-deoqamys-043YQ~s5Tay3iJ5QRI~Lxg"
+            >Slack</a
+          >
+        </div>
+        <div class="row">
+          <a
+            class="content footer-links"
+            href="https://org-roam.discourse.group/"
+            >Discourse</a
+          >
+        </div>
+      </div>
+    </section>
+  </body>
 </html>

mkdocs.yml

@@ -1,40 +0,0 @@
-site_name: "Org-roam"
-repo_url: https://github.com/org-roam/org-roam/
-edit_uri: edit/master/doc/
-copyright: "Copyright (C) 2020 Jethro Kuan and contributors"
-docs_dir: doc
-extra:
-  social:
-    - type: 'slack'
-      link: 'https://join.slack.com/t/orgroam/shared_invite/zt-clh0g0tx-j8xg1kVxnrWdKt16gmSGPQ'
-nav:
-- Home: index.md
-- A Tour of Org-roam: tour.md
-- Installation: installation.md
-- Configuration: configuration.md
-- Anatomy of an Org-roam file: anatomy.md
-- The Templating System: templating.md
-- Ecosystem: ecosystem.md
-- Similar Packages: comparison.md
-- "Appendix: Note-taking Workflow": notetaking_workflow.md
-- "Appendix: Roam Protocol": roam_protocol.md
-- "Appendix: Org Export": org_export.md
-markdown_extensions:
-  - admonition
-  - pymdownx.betterem:
-      smart_enable: all
-  - pymdownx.caret
-  - pymdownx.critic
-  - pymdownx.details
-  - pymdownx.mark
-  - pymdownx.smartsymbols
-  - pymdownx.superfences
-  - pymdownx.tasklist:
-      custom_checkbox: true
-  - pymdownx.tilde
-
-theme:
-  name: 'material'
-  palette:
-    primary: 'deep purple'
-    accent: 'deep purple'

org-roam-buffer.el

@@ -5,8 +5,8 @@
 ;; Author: Jethro Kuan <jethrokuan95@gmail.com>
 ;; URL: https://github.com/org-roam/org-roam
 ;; Keywords: org-mode, roam, convenience
-;; Version: 1.1.1
+;; Version: 1.2.1
-;; Package-Requires: ((emacs "26.1") (dash "2.13") (f "0.17.2") (s "1.12.0") (org "9.3") (emacsql "3.0.0") (emacsql-sqlite "1.0.0"))
+;; Package-Requires: ((emacs "26.1") (dash "2.13") (f "0.17.2") (s "1.12.0") (org "9.3") (emacsql "3.0.0") (emacsql-sqlite3 "1.0.0"))
 
 ;; This file is NOT part of GNU Emacs.
 
@@ -41,12 +41,15 @@
 (defvar org-roam-backlinks-mode)
 (defvar org-roam-last-window)
 (defvar org-ref-cite-types) ;; in org-ref-core.el
+(defvar org-roam-mode)
 
 (declare-function org-roam-db--ensure-built   "org-roam-db")
 (declare-function org-roam--extract-ref       "org-roam")
 (declare-function org-roam--get-title-or-slug "org-roam")
 (declare-function org-roam--get-backlinks     "org-roam")
 (declare-function org-roam-backlinks-mode     "org-roam")
+(declare-function org-roam-mode               "org-roam")
+(declare-function org-roam--find-file         "org-roam")
 
 (defcustom org-roam-buffer-position 'right
   "Position of `org-roam' buffer.
@@ -81,20 +84,28 @@ Has an effect if and only if `org-roam-buffer-position' is `top' or `bottom'."
 
 (defcustom org-roam-buffer-prepare-hook '(org-roam-buffer--insert-title
                                           org-roam-buffer--insert-backlinks
-                                          org-roam-buffer--insert-citelinks)
+                                          org-roam-buffer--insert-ref-links)
   "Hook run in the `org-roam-buffer' before it is displayed."
   :type 'hook
   :group 'org-roam)
 
-(defcustom org-roam-buffer-no-delete-other-windows nil
+(defcustom org-roam-buffer-window-parameters nil
-  "The `no-delete-other-windows' parameter of the `org-roam-buffer' window.
+  "Additional window parameters for the `org-roam-buffer' side window.
-When non-nil, the window will not be closed when deleting other windows."
+For example: (setq org-roam-buffer-window-parameters '((no-other-window . t)))"
-  :type 'boolean
+  :type '(alist)
   :group 'org-roam)
 
 (defvar org-roam-buffer--current nil
   "Currently displayed file in `org-roam' buffer.")
 
+(defun org-roam-buffer--find-file (file)
+  "Open FILE in the window `org-roam' was called from."
+  (if (and org-roam-last-window (window-valid-p org-roam-last-window))
+      (progn (with-selected-window org-roam-last-window
+               (org-roam--find-file file))
+             (select-window org-roam-last-window))
+    (org-roam--find-file file)))
+
 (defun org-roam-buffer--insert-title ()
   "Insert the org-roam-buffer title."
   (insert (propertize (org-roam--get-title-or-slug
@@ -112,34 +123,31 @@ When non-nil, the window will not be closed when deleting other windows."
                                    ,wrong-type))))))
     (concat string (when (> l 1) "s"))))
 
-(defun org-roam-buffer--insert-citelinks ()
+(defun org-roam-buffer--insert-ref-links ()
-  "Insert citation backlinks for the current buffer."
+  "Insert ref backlinks for the current buffer."
-  (if-let* ((ref (with-temp-buffer
+  (when-let ((ref (cdr (with-temp-buffer
-                   (insert-buffer-substring org-roam-buffer--current)
+                        (insert-buffer-substring org-roam-buffer--current)
-                   (org-roam--extract-ref)))
+                        (org-roam--extract-ref)))))
-            (org-ref-p (require 'org-ref nil t)) ; Ensure that org-ref is present
+    (if-let* ((key-backlinks (org-roam--get-backlinks ref))
-            (key-backlinks (org-roam--get-backlinks (cdr ref)))
+              (grouped-backlinks (--group-by (nth 0 it) key-backlinks)))
-            (grouped-backlinks (--group-by (nth 0 it) key-backlinks)))
+        (progn
-      (progn
+          (insert (let ((l (length key-backlinks)))
-        (insert (let ((l (length key-backlinks)))
+                    (format "\n\n* %d %s\n"
-                  (format "\n\n* %d %s\n"
+                            l (org-roam-buffer--pluralize "Ref Backlink" l))))
-                          l (org-roam-buffer--pluralize "Cite backlink" l))))
+          (dolist (group grouped-backlinks)
-        (dolist (group grouped-backlinks)
+            (let ((file-from (car group))
-          (let ((file-from (car group))
+                  (bls (cdr group)))
-                (bls (cdr group)))
+              (insert (format "** [[file:%s][%s]]\n"
-            (insert (format "** [[file:%s][%s]]\n"
+                              file-from
-                            file-from
+                              (org-roam--get-title-or-slug file-from)))
-                            (org-roam--get-title-or-slug file-from)))
+              (dolist (backlink bls)
-            (dolist (backlink bls)
+                (pcase-let ((`(,file-from _ ,props) backlink))
-              (pcase-let ((`(,file-from _ ,props) backlink))
+                  (insert (propertize (plist-get props :content)
-                (insert (propertize
+                           'help-echo "mouse-1: visit backlinked note"
-                         (s-trim (s-replace "\n" " "
+                           'file-from file-from
-                                            (plist-get props :content)))
+                           'file-from-point (plist-get props :point)))
-                         'help-echo "mouse-1: visit backlinked note"
+                  (insert "\n\n"))))))
-                         'file-from file-from
+      (insert "\n\n* No ref backlinks!"))))
-                         'file-from-point (plist-get props :point)))
-                (insert "\n\n"))))))
-    (insert "\n\n* No cite backlinks!")))
 
 (defun org-roam-buffer--insert-backlinks ()
   "Insert the org-roam-buffer backlinks string for the current buffer."
@@ -158,13 +166,18 @@ When non-nil, the window will not be closed when deleting other windows."
                             (org-roam--get-title-or-slug file-from)))
             (dolist (backlink bls)
               (pcase-let ((`(,file-from _ ,props) backlink))
-                (insert (propertize
+                (insert "*** "
+                        (if-let ((outline (plist-get props :outline)))
+                            (string-join outline " > ")
+                          "Top")
+                        "\n"
+                        (propertize
                          (s-trim (s-replace "\n" " "
                                             (plist-get props :content)))
                          'help-echo "mouse-1: visit backlinked note"
                          'file-from file-from
-                         'file-from-point (plist-get props :point)))
+                         'file-from-point (plist-get props :point))
-                (insert "\n\n"))))))
+                        "\n\n"))))))
     (insert "\n\n* No backlinks!")))
 
 (defun org-roam-buffer-update ()
@@ -242,8 +255,7 @@ Valid states are 'visible, 'exists and 'none."
 
 (defun org-roam-buffer--get-create ()
   "Set up the `org-roam' buffer at `org-roam-buffer-position'."
-  (let ((window (get-buffer-window))
+  (let ((position
-        (position
          (if (member org-roam-buffer-position '(right left top bottom))
              org-roam-buffer-position
            (let ((text-quoting-style 'grave))
@@ -251,25 +263,39 @@ Valid states are 'visible, 'exists and 'none."
                     "Invalid org-roam-buffer-position: %s. Defaulting to \\='right"
                     org-roam-buffer-position))
            'right)))
-    (-> (get-buffer-create org-roam-buffer)
+    (save-selected-window
-        (display-buffer-in-side-window
+      (-> (get-buffer-create org-roam-buffer)
-         `((side . ,position)
+          (display-buffer-in-side-window
-           (window-parameters . ((no-delete-other-windows . ,org-roam-buffer-no-delete-other-windows)))))
+           `((side . ,position)
-        (select-window))
+             (window-parameters . ,org-roam-buffer-window-parameters)))
-    (pcase position
+          (select-window))
-      ((or 'right 'left)
+      (pcase position
-       (org-roam-buffer--set-width  (round (* (frame-width)  org-roam-buffer-width))))
+        ((or 'right 'left)
-      ((or 'top  'bottom)
+         (org-roam-buffer--set-width
-       (org-roam-buffer--set-height (round (* (frame-height) org-roam-buffer-height)))))
+          (round (* (frame-width)  org-roam-buffer-width))))
-    (select-window window)))
+        ((or 'top  'bottom)
+         (org-roam-buffer--set-height
+          (round (* (frame-height) org-roam-buffer-height))))))))
+
+(defun org-roam-buffer-activate ()
+  "Activate display of the `org-roam-buffer'."
+  (interactive)
+  (unless org-roam-mode (org-roam-mode))
+  (setq org-roam-last-window (get-buffer-window))
+  (org-roam-buffer--get-create))
+
+(defun org-roam-buffer-deactivate ()
+  "Deactivate display of the `org-roam-buffer'."
+  (interactive)
+  (setq org-roam-last-window (get-buffer-window))
+  (delete-window (get-buffer-window org-roam-buffer)))
 
 (defun org-roam-buffer-toggle-display ()
   "Toggle display of the `org-roam-buffer'."
   (interactive)
-  (setq org-roam-last-window (get-buffer-window))
   (pcase (org-roam-buffer--visibility)
-    ('visible (delete-window (get-buffer-window org-roam-buffer)))
+    ('visible (org-roam-buffer-deactivate))
-    ((or 'exists 'none) (org-roam-buffer--get-create))))
+    ((or 'exists 'none) (org-roam-buffer-activate))))
 
 (provide 'org-roam-buffer)
 

org-roam-capture.el

@@ -5,8 +5,8 @@
 ;; Author: Jethro Kuan <jethrokuan95@gmail.com>
 ;; URL: https://github.com/org-roam/org-roam
 ;; Keywords: org-mode, roam, convenience
-;; Version: 1.1.1
+;; Version: 1.2.1
-;; Package-Requires: ((emacs "26.1") (dash "2.13") (f "0.17.2") (s "1.12.0") (org "9.3") (emacsql "3.0.0") (emacsql-sqlite "1.0.0"))
+;; Package-Requires: ((emacs "26.1") (dash "2.13") (f "0.17.2") (s "1.12.0") (org "9.3") (emacsql "3.0.0") (emacsql-sqlite3 "1.0.0"))
 
 ;; This file is NOT part of GNU Emacs.
 
@@ -31,6 +31,7 @@
 ;;; Code:
 ;;;; Library Requires
 (require 'org-capture)
+(require 'org-roam-macs)
 (require 'dash)
 (require 's)
 (require 'cl-lib)
@@ -38,17 +39,20 @@
 ;; Declarations
 (defvar org-roam-encrypt-files)
 (defvar org-roam-directory)
+(defvar org-roam-mode)
+(defvar org-roam-title-to-slug-function)
 (declare-function  org-roam--get-title-path-completions "org-roam")
 (declare-function  org-roam--get-ref-path-completions   "org-roam")
 (declare-function  org-roam--file-path-from-id          "org-roam")
+(declare-function  org-roam--find-file                  "org-roam")
 (declare-function  org-roam--format-link                "org-roam")
-(declare-function  org-roam--title-to-slug              "org-roam")
+(declare-function  org-roam-mode                        "org-roam")
 (declare-function  org-roam-completion--completing-read "org-roam-completion")
 
 (defvar org-roam-capture--file-name-default "%<%Y%m%d%H%M%S>"
   "The default file name format for Org-roam templates.")
 
-(defvar org-roam-capture--header-default "#+TITLE: ${title}\n"
+(defvar org-roam-capture--header-default "#+title: ${title}\n"
   "The default capture header for Org-roam templates.")
 
 (defvar org-roam-capture--file-path nil
@@ -79,68 +83,286 @@ note with the given `ref'.")
 (defconst org-roam-capture--template-keywords '(:file-name :head)
   "Keywords used in `org-roam-capture-templates' specific to Org-roam.")
 
-(defvar org-roam-capture-templates
+(defcustom org-roam-capture-templates
   '(("d" "default" plain (function org-roam-capture--get-point)
      "%?"
      :file-name "%<%Y%m%d%H%M%S>-${slug}"
-     :head "#+TITLE: ${title}\n"
+     :head "#+title: ${title}\n"
      :unnarrowed t))
   "Capture templates for Org-roam.
-The capture templates are an extension of
+The Org-roam capture-templates  builds on the default behaviours of
-`org-capture-templates', and the documentation there also
+`org-capture-templates' by expanding them in 3 areas:
-applies.
 
-`org-capture-templates' are extended in 3 ways:
+1. Template-expansion capabilities are extended with additional
+   custom syntax. See `org-roam-capture--fill-template' for more
+   details.
 
-1. Template expansion capabilities are extended with additional custom syntax.
+2. The `:file-name' key is added, which defines the naming format
-   See `org-roam-capture--fill-template' for more details.
+   to use when creating new notes. This file-name is relative to
-
+   `org-roam-directory', and is without the file-extension.
-2. The `:file-name' key is added, which expands to the file-name
-   of the note if it creates a new file.  This file-name is
-   relative to `org-roam-directory', and is without the
-   file-extension.
 
 3. The `:head' key is added, which contains the template that is
-   inserted on initial creation (added only once).  This is where
+   inserted upon the creation of a new file. This is where you
-   insertion of any note metadata should go.")
+   your note metadata should go.
 
-(defvar org-roam-capture-ref-templates
+Each template should have the following structure:
+
+\(KEY DESCRIPTION `plain' `(function org-roam-capture--get-point)'
+  TEMPLATE
+  `:file-name' FILENAME-FORMAT
+  `:head' HEADER-FORMAT
+  `:unnarrowed t'
+  OPTIONS-PLIST)
+
+The elements of a template-entry and their placement are the same
+as in `org-capture-templates', except that the entry type must
+always be the symbol `plain', and that the target must always be
+the list `(function org-roam-capture--get-point)'.
+
+Org-roam requires the plist elements `:file-name' and `:head' to
+be present, and it’s recommended that `:unnarrowed' be set to t."
+  :group 'org-roam
+  ;; Adapted from `org-capture-templates'
+  :type
+  '(repeat
+    (choice :value ("d" "default" plain (function org-roam-capture--get-point)
+                    "%?"
+                    :file-name "%<%Y%m%d%H%M%S>-${slug}"
+                    :head "#+title: ${title}\n"
+                    :unnarrowed t)
+            (list :tag "Multikey description"
+                  (string :tag "Keys       ")
+                  (string :tag "Description"))
+            (list :tag "Template entry"
+                  (string :tag "Keys              ")
+                  (string :tag "Description       ")
+                  (const :format "" plain)
+                  (const :format "" (function org-roam-capture--get-point))
+                  (choice :tag "Template          "
+                          (string :tag "String"
+                                  :format "String:\n            \
+Template string   :\n%v")
+                          (list :tag "File"
+                                (const :format "" file)
+                                (file :tag "Template file     "))
+                          (list :tag "Function"
+                                (const :format "" function)
+                                (function :tag "Template function ")))
+                  (const :format "File name format  :" :file-name)
+                  (string :format " %v" :value "#+title: ${title}\n")
+                  (const :format "Header format     :" :head)
+                  (string :format "\n%v" :value "%<%Y%m%d%H%M%S>-${slug}")
+                  (const :format "" :unnarrowed) (const :format "" t)
+                  (plist :inline t
+                         :tag "Options"
+                         ;; Give the most common options as checkboxes
+                         :options
+                         (((const :format "%v " :prepend) (const t))
+                          ((const :format "%v " :immediate-finish) (const t))
+                          ((const :format "%v " :jump-to-captured) (const t))
+                          ((const :format "%v " :empty-lines) (const 1))
+                          ((const :format "%v " :empty-lines-before) (const 1))
+                          ((const :format "%v " :empty-lines-after) (const 1))
+                          ((const :format "%v " :clock-in) (const t))
+                          ((const :format "%v " :clock-keep) (const t))
+                          ((const :format "%v " :clock-resume) (const t))
+                          ((const :format "%v " :time-prompt) (const t))
+                          ((const :format "%v " :tree-type) (const week))
+                          ((const :format "%v " :table-line-pos) (string))
+                          ((const :format "%v " :kill-buffer) (const t))))))))
+
+(defcustom org-roam-capture-immediate-template
+  (append (car org-roam-capture-templates) '(:immediate-finish t))
+  "Capture template to use for immediate captures in Org-roam.
+This is a single template, so do not enclose it into a list.
+See `org-roam-capture-templates' for details on templates."
+  :group 'org-roam
+  ;; Adapted from `org-capture-templates'
+  :type
+  '(list :tag "Template entry"
+         :value ("d" "default" plain (function org-roam-capture--get-point)
+                 "%?"
+                 :file-name "%<%Y%m%d%H%M%S>-${slug}"
+                 :head "#+title: ${title}\n"
+                 :unnarrowed t
+                 :immediate-finish t)
+         (string :tag "Keys              ")
+         (string :tag "Description       ")
+         (const :format "" plain)
+         (const :format "" (function org-roam-capture--get-point))
+         (choice :tag "Template          "
+                 (string :tag "String"
+                         :format "String:\n            \
+Template string   :\n%v")
+                 (list :tag "File"
+                       (const :format "" file)
+                       (file :tag "Template file     "))
+                 (list :tag "Function"
+                       (const :format "" function)
+                       (function :tag "Template function ")))
+         (const :format "File name format  :" :file-name)
+         (string :format " %v" :value "#+title: ${title}\n")
+         (const :format "Header format     :" :head)
+         (string :format "\n%v" :value "%<%Y%m%d%H%M%S>-${slug}")
+         (const :format "" :unnarrowed) (const :format "" t)
+         (const :format "" :immediate-finish) (const :format "" t)
+         (plist :inline t
+                :tag "Options"
+                ;; Give the most common options as checkboxes
+                :options
+                (((const :format "%v " :prepend) (const t))
+                 ((const :format "%v " :jump-to-captured) (const t))
+                 ((const :format "%v " :empty-lines) (const 1))
+                 ((const :format "%v " :empty-lines-before) (const 1))
+                 ((const :format "%v " :empty-lines-after) (const 1))
+                 ((const :format "%v " :clock-in) (const t))
+                 ((const :format "%v " :clock-keep) (const t))
+                 ((const :format "%v " :clock-resume) (const t))
+                 ((const :format "%v " :time-prompt) (const t))
+                 ((const :format "%v " :tree-type) (const week))
+                 ((const :format "%v " :table-line-pos) (string))
+                 ((const :format "%v " :kill-buffer) (const t))))))
+
+(defcustom org-roam-capture-ref-templates
   '(("r" "ref" plain (function org-roam-capture--get-point)
-     ""
+     "%?"
      :file-name "${slug}"
-     :head "#+TITLE: ${title}
+     :head "#+title: ${title}\n#+roam_key: ${ref}\n"
-#+ROAM_KEY: ${ref}\n"
      :unnarrowed t))
   "The Org-roam templates used during a capture from the roam-ref protocol.
-Details on how to specify for the template is given in `org-roam-capture-templates'.")
+Details on how to specify for the template is given in `org-roam-capture-templates'."
+  :group 'org-roam
+  ;; Adapted from `org-capture-templates'
+  :type
+  '(repeat
+    (choice :value ("d" "default" plain (function org-roam-capture--get-point)
+                    "%?"
+                    :file-name "${slug}"
+                    :head "#+title: ${title}\n#+roam_key: ${ref}\n"
+                    :unnarrowed t)
+            (list :tag "Multikey description"
+                  (string :tag "Keys       ")
+                  (string :tag "Description"))
+            (list :tag "Template entry"
+                  (string :tag "Keys              ")
+                  (string :tag "Description       ")
+                  (const :format "" plain)
+                  (const :format "" (function org-roam-capture--get-point))
+                  (choice :tag "Template          "
+                          (string :tag "String"
+                                  :format "String:\n            \
+Template string   :\n%v")
+                          (list :tag "File"
+                                (const :format "" file)
+                                (file :tag "Template file     "))
+                          (list :tag "Function"
+                                (const :format "" function)
+                                (function :tag "Template function ")))
+                  (const :format "File name format  :" :file-name)
+                  (string :format " %v" :value "#+title: ${title}\n")
+                  (const :format "Header format     :" :head)
+                  (string :format "\n%v" :value "%<%Y%m%d%H%M%S>-${slug}")
+                  (const :format "" :unnarrowed) (const :format "" t)
+                  (plist :inline t
+                         :tag "Options"
+                         ;; Give the most common options as checkboxes
+                         :options
+                         (((const :format "%v " :prepend) (const t))
+                          ((const :format "%v " :immediate-finish) (const t))
+                          ((const :format "%v " :jump-to-captured) (const t))
+                          ((const :format "%v " :empty-lines) (const 1))
+                          ((const :format "%v " :empty-lines-before) (const 1))
+                          ((const :format "%v " :empty-lines-after) (const 1))
+                          ((const :format "%v " :clock-in) (const t))
+                          ((const :format "%v " :clock-keep) (const t))
+                          ((const :format "%v " :clock-resume) (const t))
+                          ((const :format "%v " :time-prompt) (const t))
+                          ((const :format "%v " :tree-type) (const week))
+                          ((const :format "%v " :table-line-pos) (string))
+                          ((const :format "%v " :kill-buffer) (const t))))))))
+
+(defun org-roam-capture-p ()
+  "Return t if the current capture process is an Org-roam capture.
+This function is to only be called when org-capture-plist is
+valid for the capture (i.e. initialization, and finalization of
+the capture)."
+  (plist-get org-capture-plist :org-roam))
 
 (defun org-roam-capture--get (keyword)
-  "Gets the value for KEYWORD from the `org-roam-capture-template'."
+  "Get the value for KEYWORD from the `org-roam-capture-template'."
   (plist-get (plist-get org-capture-plist :org-roam) keyword))
 
 (defun org-roam-capture--put (&rest stuff)
-  "Puts properties from STUFF into the `org-roam-capture-template'."
+  "Put properties from STUFF into the `org-roam-capture-template'."
   (let ((p (plist-get org-capture-plist :org-roam)))
     (while stuff
-      (setq p (plist-put p
+      (setq p (plist-put p (pop stuff) (pop stuff))))
-                         (pop stuff) (pop stuff))))
     (setq org-capture-plist
           (plist-put org-capture-plist :org-roam p))))
 
-(defun org-roam-capture--in-process-p ()
+;; FIXME: Pending upstream patch
-  "Return non-nil if a `org-roam-capture' buffer exists."
+;; https://orgmode.org/list/87h7tv9pkm.fsf@hidden/T/#u
-  (cl-some (lambda (buffer)
+;;
-             (and (eq (buffer-local-value 'major-mode buffer)
+;; Org-capture's behaviour right now is that `org-capture-plist' is valid only
-                      'org-mode)
+;; during the initialization of the Org-capture buffer. The value of
-                  (plist-get (buffer-local-value 'org-capture-current-plist buffer)
+;; `org-capture-plist' is saved into buffer-local `org-capture-current-plist'.
-                             :org-roam)))
+;; However, the value for that particular capture is no longer accessible for
-           (buffer-list)))
+;; hooks in `org-capture-after-finalize-hook', since the capture buffer has been
+;; cleaned up.
+;;
+;; This advice restores the global `org-capture-plist' during finalization, so
+;; the plist is valid during both initialization and finalization of the
+;; capture.
+(defun org-roam-capture--update-plist (&optional _)
+  "Update global plist from local var."
+  (setq org-capture-plist org-capture-current-plist))
 
-(defun org-roam-capture--fill-template (str &optional info)
+(advice-add 'org-capture-finalize :before #'org-roam-capture--update-plist)
-  "Expands the template STR, returning the string.
+
+(defun org-roam-capture--finalize ()
+  "Finalize the `org-roam-capture' process."
+  (let* ((finalize (org-roam-capture--get :finalize))
+         ;; In case any regions were shielded before, unshield them
+         (region (when-let ((region (org-roam-capture--get :region)))
+                   (org-roam-unshield-region (car region) (cdr region))))
+         (beg (car region))
+         (end (cdr region)))
+    (unless org-note-abort
+      (pcase finalize
+        ('find-file
+         (when-let ((file-path (org-roam-capture--get :file-path)))
+           (org-roam--find-file file-path)
+           (run-hooks 'org-roam-capture-after-find-file-hook)))
+        ('insert-link
+         (when-let* ((mkr (org-roam-capture--get :insert-at))
+                     (buf (marker-buffer mkr)))
+           (with-current-buffer buf
+             (when region
+               (delete-region (car region) (cdr region)))
+             (let ((path (org-roam-capture--get :file-path))
+                   (desc (org-roam-capture--get :link-description)))
+               (if (eq (point) (marker-position mkr))
+                   (insert (org-roam--format-link path desc))
+                 (org-with-point-at mkr
+                   (insert (org-roam--format-link path desc))))))))))
+    (when region
+      (set-marker beg nil)
+      (set-marker end nil))
+    (org-roam-capture--save-file-maybe)
+    (remove-hook 'org-capture-after-finalize-hook #'org-roam-capture--finalize)))
+
+(defun org-roam-capture--install-finalize ()
+  "Install `org-roam-capture--finalize' if the capture is an Org-roam capture."
+  (when (org-roam-capture-p)
+    (add-hook 'org-capture-after-finalize-hook #'org-roam-capture--finalize)))
+
+(add-hook 'org-capture-prepare-finalize-hook #'org-roam-capture--install-finalize)
+
+(defun org-roam-capture--fill-template (str)
+  "Expand the template STR, returning the string.
 This is an extension of org-capture's template expansion.
 
-First, it expands ${var} occurrences in STR, using the INFO alist.
+First, it expands ${var} occurrences in STR, using `org-roam-capture--info'.
 If there is a ${var} with no matching var in the alist, the value
 of var is prompted for via `completing-read'.
 
@@ -148,23 +370,13 @@ Next, it expands the remaining template string using
 `org-capture-fill-template'."
   (-> str
       (s-format (lambda (key)
-                  (or (s--aget info key)
+                  (or (s--aget org-roam-capture--info key)
-                      (completing-read (format "%s: " key ) nil))) nil)
+                      (when-let ((val (completing-read (format "%s: " key) nil)))
+                        (push (cons key val) org-roam-capture--info)
+                        val))) nil)
       (org-capture-fill-template)))
 
-(defun org-roam-capture--insert-link-h ()
+(defun org-roam-capture--save-file-maybe ()
-  "Insert the link into the original buffer, after the capture process is done.
-This is added as a hook to `org-capture-after-finalize-hook'."
-  (when (and (not org-note-abort)
-             (eq (org-roam-capture--get :capture-fn)
-                 'org-roam-insert))
-    (when-let ((region (org-roam-capture--get :region))) ;; Remove previously selected text.
-      (delete-region (car region) (cdr region)))
-    (insert (org-roam--format-link (org-roam-capture--get :file-path)
-                                   (org-roam-capture--get :link-description))))
-  (remove-hook 'org-capture-after-finalize-hook #'org-roam-capture--insert-link-h))
-
-(defun org-roam-capture--save-file-maybe-h ()
   "Save the file conditionally.
 The file is saved if the original value of :no-save is not t and
 `org-note-abort' is not t. It is added to
@@ -178,8 +390,7 @@ The file is saved if the original value of :no-save is not t and
    ((and (not (org-roam-capture--get :orig-no-save))
          (not org-note-abort))
     (with-current-buffer (org-capture-get :buffer)
-      (save-buffer))))
+      (save-buffer)))))
-  (remove-hook 'org-capture-after-finalize-hook #'org-roam-capture--save-file-maybe-h))
 
 (defun org-roam-capture--new-file ()
   "Return the path to the new file during an Org-roam capture.
@@ -199,14 +410,13 @@ aborted, we do the following:
 
 2. Set the capture template's :no-save to t.
 
-3. Add a function on `org-capture-after-finalize-hook' that saves
+3. Add a function on `org-capture-before-finalize-hook' that saves
 the file if the original value of :no-save is not t and
 `org-note-abort' is not t."
   (let* ((name-templ (or (org-roam-capture--get :file-name)
                          org-roam-capture--file-name-default))
          (new-id (s-trim (org-roam-capture--fill-template
-                          name-templ
+                          name-templ)))
-                          org-roam-capture--info)))
          (file-path (org-roam--file-path-from-id new-id))
          (roam-head (or (org-roam-capture--get :head)
                         org-roam-capture--header-default))
@@ -227,16 +437,6 @@ the file if the original value of :no-save is not t and
                        :no-save t))
     file-path))
 
-(defun org-roam-capture--expand-template ()
-  "Expand capture template with information from `org-roam-capture--info'."
-  (org-capture-put :template
-                   (s-format (org-capture-get :template)
-                             (lambda (key)
-                               (or (s--aget org-roam-capture--info key)
-                                   (when-let ((v (completing-read (format "%s: " key ) nil)))
-                                     (push (cons key v) org-roam-capture--info)
-                                     v))) nil)))
-
 (defun org-roam-capture--get-point ()
   "Return exact point to file for org-capture-template.
 The file to use is dependent on the context:
@@ -265,10 +465,12 @@ This function is used solely in Org-roam's capture templates: see
                      ('ref
                       (let ((completions (org-roam--get-ref-path-completions))
                             (ref (cdr (assoc 'ref org-roam-capture--info))))
-                        (or (cdr (assoc ref completions))
+                        (if-let ((pl (cdr (assoc ref completions))))
-                            (org-roam-capture--new-file))))
+                            (plist-get pl :path)
+                          (org-roam-capture--new-file))))
                      (_ (error "Invalid org-roam-capture-context")))))
-    (org-roam-capture--expand-template)
+    (org-capture-put :template
+                     (org-roam-capture--fill-template (org-capture-get :template)))
     (org-roam-capture--put :file-path file-path)
     (while org-roam-capture-additional-template-props
       (let ((prop (pop org-roam-capture-additional-template-props))
@@ -296,52 +498,48 @@ This function is used solely in Org-roam's capture templates: see
        (append converted options `(:org-roam ,org-roam-plist))))
     (_ (user-error "Invalid capture template format: %s" template))))
 
-(defun org-roam-capture--find-file-h ()
-  "Opens the newly created template file.
-This is added as a hook to `org-capture-after-finalize-hook'.
-Run the hooks defined in `org-roam-capture-after-find-file-hook'."
-  (unless org-note-abort
-    (when-let ((file-path (org-roam-capture--get :file-path)))
-      (find-file file-path))
-    (run-hooks 'org-roam-capture-after-find-file-hook))
-  (remove-hook 'org-capture-after-finalize-hook #'org-roam-capture--find-file-h))
-
 (defcustom org-roam-capture-after-find-file-hook nil
   "Hook that is run right after an Org-roam capture process is finalized.
 Suitable for moving point."
   :group 'org-roam
   :type 'hook)
 
+(defcustom org-roam-capture-function #'org-capture
+  "Function that is invoked to start the `org-capture' process."
+  :group 'org-roam
+  :type 'function)
+
 (defun org-roam-capture--capture (&optional goto keys)
   "Create a new file, and return the path to the edited file.
 The templates are defined at `org-roam-capture-templates'.  The
 GOTO and KEYS argument have the same functionality as
 `org-capture'."
-  (let ((org-capture-templates (mapcar #'org-roam-capture--convert-template org-roam-capture-templates))
+  (let* ((org-capture-templates (mapcar #'org-roam-capture--convert-template org-roam-capture-templates))
-        org-capture-templates-contexts)
+         (one-template-p (= (length org-capture-templates) 1))
-    (when (= (length org-capture-templates) 1)
+         org-capture-templates-contexts)
+    (when one-template-p
       (setq keys (caar org-capture-templates)))
-    (add-hook 'org-capture-after-finalize-hook #'org-roam-capture--save-file-maybe-h)
+    (if (or one-template-p
-    (org-capture goto keys)))
+            (eq org-roam-capture-function 'org-capture))
+        (org-capture goto keys)
+      (funcall-interactively org-roam-capture-function))))
 
 ;;;###autoload
 (defun org-roam-capture ()
   "Launches an `org-capture' process for a new or existing note.
 This uses the templates defined at `org-roam-capture-templates'."
   (interactive)
-  (when (org-roam-capture--in-process-p)
+  (unless org-roam-mode (org-roam-mode))
-    (user-error "Nested Org-roam capture processes not supported"))
   (let* ((completions (org-roam--get-title-path-completions))
          (title-with-keys (org-roam-completion--completing-read "File: "
                                                                 completions))
          (res (cdr (assoc title-with-keys completions)))
          (title (or (plist-get res :title) title-with-keys))
-         (file-path (plist-get res :file-path)))
+         (file-path (plist-get res :path)))
     (let ((org-roam-capture--info (list (cons 'title title)
-                                        (cons 'slug (org-roam--title-to-slug title))
+                                        (cons 'slug (funcall org-roam-title-to-slug-function title))
                                         (cons 'file file-path)))
           (org-roam-capture--context 'capture))
-      (setq org-roam-capture-additional-template-props (list :capture-fn 'org-roam-capture))
       (condition-case err
           (org-roam-capture--capture)
         (error (user-error "%s.  Please adjust `org-roam-capture-templates'"

org-roam-compat.el

@@ -5,8 +5,8 @@
 ;; Author: Jethro Kuan <jethrokuan95@gmail.com>
 ;; URL: https://github.com/org-roam/org-roam
 ;; Keywords: org-mode, roam, convenience
-;; Version: 1.1.1
+;; Version: 1.2.1
-;; Package-Requires: ((emacs "26.1") (dash "2.13") (f "0.17.2") (s "1.12.0") (org "9.3") (emacsql "3.0.0") (emacsql-sqlite "1.0.0"))
+;; Package-Requires: ((emacs "26.1") (dash "2.13") (f "0.17.2") (s "1.12.0") (org "9.3") (emacsql "3.0.0") (emacsql-sqlite3 "1.0.0"))
 
 ;; This file is NOT part of GNU Emacs.
 
@@ -79,6 +79,11 @@
   "org-roam 1.1.0")
 (define-obsolete-function-alias 'org-roam-db--maybe-update 'org-roam-db--update-maybe
   "org-roam 1.1.0")
+(define-obsolete-function-alias 'org-roam-db--clear 'org-roam-db-clear
+  "org-roam 1.2.0")
+
+(when (version< (org-version) "9.3")
+  (defalias 'org-link-make-string 'org-make-link-string))
 
 ;;;; Variables
 (define-obsolete-variable-alias 'org-roam-graphviz-extra-options
@@ -95,6 +100,8 @@
   'org-roam-dailies-capture-templates "org-roam 1.0.0")
 (define-obsolete-variable-alias 'org-roam-date-filename-format
   'org-roam-dailies-capture-templates "org-roam 1.0.0")
+(make-obsolete-variable 'org-roam-buffer-no-delete-other-windows
+  'org-roam-buffer-window-parameters "org-roam 1.1.1")
 
 (provide 'org-roam-compat)
 

org-roam-completion.el

@@ -5,8 +5,8 @@
 ;; Author: Jethro Kuan <jethrokuan95@gmail.com>
 ;; URL: https://github.com/org-roam/org-roam
 ;; Keywords: org-mode, roam, convenience
-;; Version: 1.1.1
+;; Version: 1.2.1
-;; Package-Requires: ((emacs "26.1") (dash "2.13") (f "0.17.2") (s "1.12.0") (org "9.3") (emacsql "3.0.0") (emacsql-sqlite "1.0.0"))
+;; Package-Requires: ((emacs "26.1") (dash "2.13") (f "0.17.2") (s "1.12.0") (org "9.3") (emacsql "3.0.0") (emacsql-sqlite3 "1.0.0"))
 
 ;; This file is NOT part of GNU Emacs.
 

org-roam-dailies.el

@@ -5,8 +5,8 @@
 ;; Author: Jethro Kuan <jethrokuan95@gmail.com>
 ;; URL: https://github.com/org-roam/org-roam
 ;; Keywords: org-mode, roam, convenience
-;; Version: 1.1.1
+;; Version: 1.2.1
-;; Package-Requires: ((emacs "26.1") (dash "2.13") (f "0.17.2") (s "1.12.0") (org "9.3") (emacsql "3.0.0") (emacsql-sqlite "1.0.0"))
+;; Package-Requires: ((emacs "26.1") (dash "2.13") (f "0.17.2") (s "1.12.0") (org "9.3") (emacsql "3.0.0") (emacsql-sqlite3 "1.0.0"))
 
 ;; This file is NOT part of GNU Emacs.
 
@@ -36,41 +36,95 @@
 (require 'org-roam-capture)
 (require 'org-roam-macs)
 
-(defvar org-roam-dailies-capture-templates
+(defcustom org-roam-dailies-capture-templates
   '(("d" "daily" plain (function org-roam-capture--get-point)
      ""
      :immediate-finish t
      :file-name "%<%Y-%m-%d>"
-     :head "#+TITLE: %<%Y-%m-%d>"))
+     :head "#+title: %<%Y-%m-%d>"))
-  "Capture templates for daily notes in Org-roam.")
+  "Capture templates for daily notes in Org-roam."
+  :group 'org-roam
+  ;; Adapted from `org-capture-templates'
+  :type
+  '(repeat
+    (choice :value ("d" "daily" plain (function org-roam-capture--get-point)
+                    ""
+                    :immediate-finish t
+                    :file-name "%<%Y-%m-%d>"
+                    :head "#+title: %<%Y-%m-%d>")
+            (list :tag "Multikey description"
+                  (string :tag "Keys       ")
+                  (string :tag "Description"))
+            (list :tag "Template entry"
+                  (string :tag "Keys              ")
+                  (string :tag "Description       ")
+                  (const :format "" plain)
+                  (const :format "" (function org-roam-capture--get-point))
+                  (choice :tag "Template          "
+                          (string :tag "String"
+                                  :format "String:\n            \
+Template string   :\n%v")
+                          (list :tag "File"
+                                (const :format "" file)
+                                (file :tag "Template file     "))
+                          (list :tag "Function"
+                                (const :format "" function)
+                                (function :tag "Template function ")))
+                  (const :format "" :immediate-finish) (const :format "" t)
+                  (const :format "File name format  :" :file-name)
+                  (string :format " %v" :value "#+title: ${title}\n")
+                  (const :format "Header format     :" :head)
+                  (string :format "\n%v" :value "%<%Y%m%d%H%M%S>-${slug}")
+                  (plist :inline t
+                         :tag "Options"
+                         ;; Give the most common options as checkboxes
+                         :options
+                         (((const :format "%v " :prepend) (const t))
+                          ((const :format "%v " :jump-to-captured) (const t))
+                          ((const :format "%v " :empty-lines) (const 1))
+                          ((const :format "%v " :empty-lines-before) (const 1))
+                          ((const :format "%v " :empty-lines-after) (const 1))
+                          ((const :format "%v " :clock-in) (const t))
+                          ((const :format "%v " :clock-keep) (const t))
+                          ((const :format "%v " :clock-resume) (const t))
+                          ((const :format "%v " :time-prompt) (const t))
+                          ((const :format "%v " :tree-type) (const week))
+                          ((const :format "%v " :table-line-pos) (string))
+                          ((const :format "%v " :kill-buffer) (const t))
+                          ((const :format "%v " :unnarrowed) (const t))))))))
 
 ;; Declarations
+(defvar org-roam-mode)
 (declare-function org-roam--file-path-from-id "org-roam")
+(declare-function org-roam-mode               "org-roam")
 
 (defun org-roam-dailies--file-for-time (time)
   "Create and find file for TIME."
   (let ((org-roam-capture-templates org-roam-dailies-capture-templates)
         (org-roam-capture--info (list (cons 'time time)))
         (org-roam-capture--context 'dailies))
-    (add-hook 'org-capture-after-finalize-hook #'org-roam-capture--find-file-h)
+    (setq org-roam-capture-additional-template-props (list :finalize 'find-file))
     (org-roam--with-template-error 'org-roam-dailies-capture-templates
       (org-roam-capture--capture))))
 
 (defun org-roam-dailies-today ()
   "Create and find the daily note for today."
   (interactive)
+  (unless org-roam-mode (org-roam-mode))
   (org-roam-dailies--file-for-time (current-time)))
 
 (defun org-roam-dailies-tomorrow (n)
   "Create and find the daily note for tomorrow.
 With numeric argument N, use N days in the future."
   (interactive "p")
+  (unless org-roam-mode (org-roam-mode))
   (org-roam-dailies--file-for-time (time-add (* n 86400) (current-time))))
 
 (defun org-roam-dailies-yesterday (n)
   "Create and find the file for yesterday.
 With numeric argument N, use N days in the past."
   (interactive "p")
+  (unless org-roam-mode (org-roam-mode))
   (org-roam-dailies-tomorrow (- n)))
 
 (defun org-roam-dailies-date ()

org-roam-db.el

@@ -5,8 +5,8 @@
 ;; Author: Jethro Kuan <jethrokuan95@gmail.com>
 ;; URL: https://github.com/org-roam/org-roam
 ;; Keywords: org-mode, roam, convenience
-;; Version: 1.1.1
+;; Version: 1.2.1
-;; Package-Requires: ((emacs "26.1") (dash "2.13") (f "0.17.2") (s "1.12.0") (org "9.3") (emacsql "3.0.0") (emacsql-sqlite "1.0.0"))
+;; Package-Requires: ((emacs "26.1") (dash "2.13") (f "0.17.2") (s "1.12.0") (org "9.3") (emacsql "3.0.0") (emacsql-sqlite3 "1.0.0"))
 
 ;; This file is NOT part of GNU Emacs.
 
@@ -33,23 +33,28 @@
 ;;;; Library Requires
 (eval-when-compile (require 'subr-x))
 (require 'emacsql)
-(require 'emacsql-sqlite)
+(require 'emacsql-sqlite3)
+(require 'seq)
 (require 'org-roam-macs)
 
 (defvar org-roam-directory)
 (defvar org-roam-verbose)
+(defvar org-roam-file-name)
 
 (declare-function org-roam--org-roam-file-p                "org-roam")
 (declare-function org-roam--extract-titles                 "org-roam")
 (declare-function org-roam--extract-ref                    "org-roam")
 (declare-function org-roam--extract-tags                   "org-roam")
+(declare-function org-roam--extract-headlines              "org-roam")
 (declare-function org-roam--extract-links                  "org-roam")
 (declare-function org-roam--list-all-files                 "org-roam")
+(declare-function org-roam--path-to-slug                   "org-roam")
+(declare-function org-roam--file-name-extension            "org-roam")
 (declare-function org-roam-buffer--update-maybe            "org-roam-buffer")
 
 ;;;; Options
 (defcustom org-roam-db-location nil
-  "Location of the Org-roam database.
+  "The full path to file where the Org-roam database is stored.
 If this is non-nil, the Org-roam sqlite database is saved here.
 
 It is the user's responsibility to set this correctly, especially
@@ -57,11 +62,22 @@ when used with multiple Org-roam instances."
   :type 'string
   :group 'org-roam)
 
-(defconst org-roam-db--version 5)
+(defcustom org-roam-db-gc-threshold gc-cons-threshold
-(defconst org-roam-db--sqlite-available-p
+  "The value to temporarily set the `gc-cons-threshold' threshold to.
-  (with-demoted-errors "Org-roam initialization: %S"
+During large, heavy operations like `org-roam-db-build-cache',
-    (emacsql-sqlite-ensure-binary)
+many GC operations happen because of the large number of
-    t))
+temporary structures generated (e.g. parsed ASTs). Temporarily
+increasing `gc-cons-threshold' will help reduce the number of GC
+operations, at the cost of temporary memory usage.
+
+This defaults to the original value of `gc-cons-threshold', but
+tweaking this number may lead to better overall performance. For
+example, to reduce the number of GCs, one may set it to a large
+value like `most-positive-fixnum'."
+  :type 'int
+  :group 'org-roam)
+
+(defconst org-roam-db--version 7)
 
 (defvar org-roam-db--connection (make-hash-table :test #'equal)
   "Database connection to Org-roam database.")
@@ -69,7 +85,6 @@ when used with multiple Org-roam instances."
 ;;;; Core Functions
 (defun org-roam-db--get ()
   "Return the sqlite db file."
-  (interactive "P")
   (or org-roam-db-location
       (expand-file-name "org-roam.db" org-roam-directory)))
 
@@ -87,7 +102,7 @@ Performs a database upgrade when required."
     (let* ((db-file (org-roam-db--get))
            (init-db (not (file-exists-p db-file))))
       (make-directory (file-name-directory db-file) t)
-      (let ((conn (emacsql-sqlite db-file)))
+      (let ((conn (emacsql-sqlite3 db-file)))
         (set-process-query-on-exit-flag (emacsql-process conn) nil)
         (puthash (file-truename org-roam-directory)
                  conn
@@ -123,6 +138,10 @@ SQL can be either the emacsql vector representation, or a string."
       (hash :not-null)
       (meta :not-null)])
 
+    (headlines
+     [(id :unique :primary-key)
+      (file :not-null)])
+
     (links
      [(from :not-null)
       (to :not-null)
@@ -135,7 +154,7 @@ SQL can be either the emacsql vector representation, or a string."
 
     (titles
      [(file :not-null)
-      titles])
+      title])
 
     (refs
      [(ref :unique :not-null)
@@ -177,7 +196,7 @@ the current `org-roam-directory'."
 ;;;; Database API
 ;;;;; Initialization
 (defun org-roam-db--initialized-p ()
-  "Whether the cache has been initialized."
+  "Whether the Org-roam cache has been initialized."
   (and (file-exists-p (org-roam-db--get))
        (> (caar (org-roam-db-query [:select (funcall count) :from titles]))
           0)))
@@ -188,8 +207,8 @@ the current `org-roam-directory'."
     (error "[Org-roam] your cache isn't built yet! Please run org-roam-db-build-cache")))
 
 ;;;;; Clearing
-(defun org-roam-db--clear ()
+(defun org-roam-db-clear ()
-  "Clears all entries in the caches."
+  "Clears all entries in the Org-roam cache."
   (interactive)
   (when (file-exists-p (org-roam-db--get))
     (dolist (table (mapcar #'car org-roam-db--table-schemata))
@@ -225,7 +244,29 @@ This is equivalent to removing the node from the graph."
   (org-roam-db-query
    [:insert :into titles
     :values $v1]
-   (list (vector file titles))))
+   (mapcar (lambda (title)
+             (vector file title)) titles)))
+
+(defun org-roam-db--insert-headlines (headlines)
+  "Insert HEADLINES into the Org-roam cache.
+Returns t if the insertion was successful, nil otherwise.
+Insertions can fail when there is an ID conflict."
+  (condition-case nil
+      (progn
+        (org-roam-db-query
+           [:insert :into headlines
+            :values $v1]
+           headlines)
+        t)
+    (error
+     (unless (listp headlines)
+       (setq headlines (list headlines)))
+     (lwarn '(org-roam) :error
+            (format "Duplicate IDs in %s, one of:\n\n%s\n\nskipping..."
+                    (aref (car headlines) 1)
+                    (string-join (mapcar (lambda (hl)
+                                           (aref hl 0)) headlines) "\n")))
+     nil)))
 
 (defun org-roam-db--insert-tags (file tags)
   "Insert TAGS for a FILE into the Org-roam cache."
@@ -235,12 +276,27 @@ This is equivalent to removing the node from the graph."
    (list (vector file tags))))
 
 (defun org-roam-db--insert-ref (file ref)
-  "Insert REF for FILE into the Org-roam cache."
+  "Insert REF for FILE into the Org-roam cache.
+Returns t if successful, and nil otherwise.
+Insertions can fail if the key is already in the database."
   (let ((key (cdr ref))
         (type (car ref)))
-    (org-roam-db-query
+    (condition-case nil
-     [:insert :into refs :values $v1]
+        (progn
-     (list (vector key file type)))))
+         (org-roam-db-query
+          [:insert :into refs :values $v1]
+          (list (vector key file type)))
+         t)
+      (error
+       (lwarn '(org-roam) :error
+              (format "Duplicate ref %s in:\n\nA: %s\nB: %s\n\nskipping..."
+                      key
+                      file
+                      (caar (org-roam-db-query
+                             [:select file :from refs
+                              :where (= ref $v1)]
+                             (vector key)))))
+       nil))))
 
 ;;;;; Fetching
 (defun org-roam-db--get-current-files ()
@@ -253,22 +309,22 @@ This is equivalent to removing the node from the graph."
 
 (defun org-roam-db--get-titles (file)
   "Return the titles of FILE from the cache."
-  (caar (org-roam-db-query [:select [titles] :from titles
+  (caar (org-roam-db-query [:select [title] :from titles
-                            :where (= file $s1)]
+                            :where (= file $s1)
-                           file
+                            :limit 1]
-                           :limit 1)))
+                           file)))
 
 (defun org-roam-db--connected-component (file)
   "Return all files reachable from/connected to FILE, including the file itself.
 If the file does not have any connections, nil is returned."
   (let* ((query "WITH RECURSIVE
                    links_of(file, link) AS
-                     (WITH roamlinks AS (SELECT * FROM links WHERE \"type\" = '\"roam\"'),
+                     (WITH filelinks AS (SELECT * FROM links WHERE \"type\" = '\"file\"'),
                            citelinks AS (SELECT * FROM links
                                                   JOIN refs ON links.\"to\" = refs.\"ref\"
                                                             AND links.\"type\" = '\"cite\"')
-                      SELECT \"from\", \"to\" FROM roamlinks UNION
+                      SELECT \"from\", \"to\" FROM filelinks UNION
-                      SELECT \"to\", \"from\" FROM roamlinks UNION
+                      SELECT \"to\", \"from\" FROM filelinks UNION
                       SELECT \"file\", \"from\" FROM citelinks UNION
                       SELECT \"from\", \"file\" FROM citelinks),
                    connected_component(file) AS
@@ -285,12 +341,12 @@ This includes the file itself. If the file does not have any
 connections, nil is returned."
   (let* ((query "WITH RECURSIVE
                    links_of(file, link) AS
-                     (WITH roamlinks AS (SELECT * FROM links WHERE \"type\" = '\"roam\"'),
+                     (WITH filelinks AS (SELECT * FROM links WHERE \"type\" = '\"file\"'),
                            citelinks AS (SELECT * FROM links
                                                   JOIN refs ON links.\"to\" = refs.\"ref\"
                                                             AND links.\"type\" = '\"cite\"')
-                      SELECT \"from\", \"to\" FROM roamlinks UNION
+                      SELECT \"from\", \"to\" FROM filelinks UNION
-                      SELECT \"to\", \"from\" FROM roamlinks UNION
+                      SELECT \"to\", \"from\" FROM filelinks UNION
                       SELECT \"file\", \"from\" FROM citelinks UNION
                       SELECT \"from\", \"file\" FROM citelinks),
                    -- Links are traversed in a breadth-first search.  In order to calculate the
@@ -312,6 +368,26 @@ connections, nil is returned."
          (files (mapcar 'car-safe (emacsql (org-roam-db) query file max-distance))))
     files))
 
+(defun org-roam-db--file-hash (&optional file-path)
+  "Compute the hash of FILE-PATH, a file or current buffer."
+  (let* ((file-p (and file-path))
+         (file-path (or file-path
+                        (buffer-file-name (current-buffer))))
+         (encrypted-p (and file-path
+                           (string= (org-roam--file-name-extension file-path)
+                                    "gpg"))))
+    (cond ((and encrypted-p file-p)
+           (with-temp-buffer
+             (set-buffer-multibyte nil)
+             (insert-file-contents-literally file-path)
+             (secure-hash 'sha1 (current-buffer))))
+          (file-p
+           (with-temp-buffer
+             (insert-file-contents file-path)
+             (secure-hash 'sha1 (current-buffer))))
+          (t
+           (secure-hash 'sha1 (current-buffer))))))
+
 ;;;;; Updating
 (defun org-roam-db--update-meta ()
   "Update the metadata of the current buffer into the cache."
@@ -319,7 +395,7 @@ connections, nil is returned."
          (attr (file-attributes file))
          (atime (file-attribute-access-time attr))
          (mtime (file-attribute-modification-time attr))
-         (hash (secure-hash 'sha1 (current-buffer))))
+         (hash (org-roam-db--file-hash)))
     (org-roam-db-query [:delete :from files
                         :where (= file $s1)]
                        file)
@@ -328,20 +404,22 @@ connections, nil is returned."
 (defun org-roam-db--update-titles ()
   "Update the title of the current buffer into the cache."
   (let* ((file (file-truename (buffer-file-name)))
-         (title (org-roam--extract-titles)))
+         (titles (or (org-roam--extract-titles)
+                     (list (org-roam--path-to-slug file)))))
     (org-roam-db-query [:delete :from titles
                         :where (= file $s1)]
                        file)
-    (org-roam-db--insert-titles file title)))
+    (org-roam-db--insert-titles file titles)))
 
 (defun org-roam-db--update-tags ()
   "Update the tags of the current buffer into the cache."
-  (let* ((file (file-truename (buffer-file-name)))
+  (let ((file (file-truename (buffer-file-name)))
-         (tags (org-roam--extract-tags)))
+        (tags (org-roam--extract-tags)))
     (org-roam-db-query [:delete :from tags
                         :where (= file $s1)]
                        file)
-    (org-roam-db--insert-tags file tags)))
+    (when tags
+      (org-roam-db--insert-tags file tags))))
 
 (defun org-roam-db--update-refs ()
   "Update the ref of the current buffer into the cache."
@@ -352,7 +430,7 @@ connections, nil is returned."
     (when-let ((ref (org-roam--extract-ref)))
       (org-roam-db--insert-ref file ref))))
 
-(defun org-roam-db--update-cache-links ()
+(defun org-roam-db--update-links ()
   "Update the file links of the current buffer in the cache."
   (let ((file (file-truename (buffer-file-name))))
     (org-roam-db-query [:delete :from links
@@ -361,6 +439,15 @@ connections, nil is returned."
     (when-let ((links (org-roam--extract-links)))
       (org-roam-db--insert-links links))))
 
+(defun org-roam-db--update-headlines ()
+  "Update the file headlines of the current buffer into the cache."
+  (let* ((file (file-truename (buffer-file-name))))
+    (org-roam-db-query [:delete :from headlines
+                        :where (= file $s1)]
+                       file)
+    (when-let ((headlines (org-roam--extract-headlines)))
+      (org-roam-db--insert-headlines headlines))))
+
 (defun org-roam-db--update-file (&optional file-path)
   "Update Org-roam cache for FILE-PATH."
   (when (org-roam--org-roam-file-p file-path)
@@ -369,11 +456,13 @@ connections, nil is returned."
                    (current-buffer))))
       (with-current-buffer buf
         (save-excursion
-          (org-roam-db--update-meta)
+          (emacsql-with-transaction (org-roam-db)
-          (org-roam-db--update-tags)
+            (org-roam-db--update-meta)
-          (org-roam-db--update-titles)
+            (org-roam-db--update-tags)
-          (org-roam-db--update-refs)
+            (org-roam-db--update-titles)
-          (org-roam-db--update-cache-links)
+            (org-roam-db--update-refs)
+            (org-roam-db--update-headlines)
+            (org-roam-db--update-links))
           (org-roam-buffer--update-maybe :redisplay t))))))
 
 (defun org-roam-db-build-cache (&optional force)
@@ -383,75 +472,80 @@ If FORCE, force a rebuild of the cache from scratch."
   (when force (delete-file (org-roam-db--get)))
   (org-roam-db--close) ;; Force a reconnect
   (org-roam-db) ;; To initialize the database, no-op if already initialized
-  (let* ((org-roam-files (org-roam--list-all-files))
+  (let* ((gc-cons-threshold org-roam-db-gc-threshold)
+         (org-roam-files (org-roam--list-all-files))
          (current-files (org-roam-db--get-current-files))
-         all-files all-links all-titles all-refs all-tags)
+         (file-count 0)
-    (dolist (file org-roam-files)
+         (headline-count 0)
-      (let* ((attr (file-attributes file))
+         (link-count 0)
-             (atime (file-attribute-access-time attr))
+         (tag-count 0)
-             (mtime (file-attribute-modification-time attr)))
+         (title-count 0)
-        (org-roam--with-temp-buffer
+         (ref-count 0)
-          (insert-file-contents file)
+         (deleted-count 0))
-          (let ((contents-hash (secure-hash 'sha1 (current-buffer))))
+    (emacsql-with-transaction (org-roam-db)
+      ;; Two-step building
+      ;; First step: Rebuild files and headlines
+      (dolist (file org-roam-files)
+        (let* ((attr (file-attributes file))
+               (atime (file-attribute-access-time attr))
+               (mtime (file-attribute-modification-time attr)))
+          (let ((contents-hash (org-roam-db--file-hash file)))
             (unless (string= (gethash file current-files)
                              contents-hash)
-              (org-roam-db--clear-file file)
+              (condition-case nil
-              (push (vector file contents-hash (list :atime atime :mtime mtime))
+                  (org-roam--with-temp-buffer file
-                    all-files)
+                    (org-roam-db--clear-file file)
-              (when-let (links (org-roam--extract-links file))
+                    (org-roam-db-query
-                (push links all-links))
+                     [:insert :into files
-              (when-let (tags (org-roam--extract-tags file))
+                      :values $v1]
-                (push (vector file tags) all-tags))
+                     (vector file contents-hash (list :atime atime :mtime mtime)))
-              (let ((titles (org-roam--extract-titles)))
+                    (setq file-count (1+ file-count))
-                (push (vector file titles)
+                    (when-let ((headlines (org-roam--extract-headlines file)))
-                      all-titles))
+                      (when (org-roam-db--insert-headlines headlines)
-              (when-let* ((ref (org-roam--extract-ref))
+                        (setq headline-count (1+ headline-count)))))
-                          (type (car ref))
+                (file-error
-                          (key (cdr ref)))
+                 (setq org-roam-files (remove file org-roam-files))
-                (setq all-refs (cons (vector key file type) all-refs))))
+                 (org-roam-db--clear-file file)
-            (remhash file current-files)))))
+                 (lwarn '(org-roam) :warning
-    (dolist (file (hash-table-keys current-files))
+                        "Skipping unreadable file while building cache: %s" file)))))))
-      ;; These files are no longer around, remove from cache...
+      ;; Second step: Rebuild the rest
-      (org-roam-db--clear-file file))
+      (dolist (file org-roam-files)
-    (when all-files
+        (let ((contents-hash (org-roam-db--file-hash file)))
-      (org-roam-db-query
+          (unless (string= (gethash file current-files)
-       [:insert :into files
+                           contents-hash)
-        :values $v1]
+            (org-roam--with-temp-buffer file
-       all-files))
+             (when-let (links (org-roam--extract-links file))
-    (when all-links
+               (org-roam-db-query
-      (org-roam-db-query
+                [:insert :into links
-       [:insert :into links
+                 :values $v1]
-        :values $v1]
+                links)
-       all-links))
+               (setq link-count (1+ link-count)))
-    (when all-titles
+             (when-let (tags (org-roam--extract-tags file))
-      (org-roam-db-query
+               (org-roam-db-query
-       [:insert :into titles
+                [:insert :into tags
-        :values $v1]
+                 :values $v1]
-       all-titles))
+                (vector file tags))
-    (when all-tags
+               (setq tag-count (1+ tag-count)))
-      (org-roam-db-query
+             (let ((titles (or (org-roam--extract-titles)
-       [:insert :into tags
+                               (list (org-roam--path-to-slug file)))))
-        :values $v1]
+               (org-roam-db--insert-titles file titles)
-       all-tags))
+               (setq title-count (+ title-count (length titles))))
-    (when all-refs
+             (when-let* ((ref (org-roam--extract-ref)))
-      (org-roam-db-query
+               (when (org-roam-db--insert-ref file ref)
-       [:insert :into refs
+                 (setq ref-count (1+ ref-count))))))
-        :values $v1]
+          (remhash file current-files)))
-       all-refs))
+      (dolist (file (hash-table-keys current-files))
-    (let ((stats (list :files (length all-files)
+        ;; These files are no longer around, remove from cache...
-                       :links (length all-links)
+        (org-roam-db--clear-file file)
-                       :tags (length all-tags)
+        (setq deleted-count (1+ deleted-count))))
-                       :titles (length all-titles)
+    (org-roam-message "files: Δ%s, headlines: Δ%s, links: Δ%s, tags: Δ%s, titles: Δ%s, refs: Δ%s, deleted: Δ%s"
-                       :refs (length all-refs)
+                      file-count
-                       :deleted (length (hash-table-keys current-files)))))
+                      headline-count
-      (org-roam-message "files: %s, links: %s, tags: %s, titles: %s, refs: %s, deleted: %s"
+                      link-count
-                        (plist-get stats :files)
+                      tag-count
-                        (plist-get stats :links)
+                      title-count
-                        (plist-get stats :tags)
+                      ref-count
-                        (plist-get stats :titles)
+                      deleted-count)))
-                        (plist-get stats :refs)
-                        (plist-get stats :deleted))
-      stats)))
 
 (provide 'org-roam-db)
 

org-roam-dev.el

@@ -5,8 +5,8 @@
 ;; Author: Jethro Kuan <jethrokuan95@gmail.com>
 ;; URL: https://github.com/org-roam/org-roam
 ;; Keywords: org-mode, roam, convenience
-;; Version: 1.1.1
+;; Version: 1.2.1
-;; Package-Requires: ((emacs "26.1") (dash "2.13") (f "0.17.2") (s "1.12.0") (org "9.3") (emacsql "3.0.0") (emacsql-sqlite "1.0.0"))
+;; Package-Requires: ((emacs "26.1") (dash "2.13") (f "0.17.2") (s "1.12.0") (org "9.3") (emacsql "3.0.0") (emacsql-sqlite3 "1.0.0"))
 
 ;; This file is NOT part of GNU Emacs.
 
@@ -33,8 +33,14 @@
 ;;
 ;;; Code:
 (require 'emacsql)
-(emacsql-fix-vector-indentation)
-(setq-local sentence-end-double-space nil)
-(provide 'org-roam-dev)
 
+;;;###autoload
+(define-minor-mode org-roam-dev-mode
+  "Minor mode for setting the dev environment of Org-roam."
+  :lighter " ORD"
+  (when org-roam-dev-mode
+    (emacsql-fix-vector-indentation)
+    (setq-local sentence-end-double-space nil)))
+
+(provide 'org-roam-dev)
 ;;; org-roam-dev.el ends here

org-roam-doctor.el

@@ -5,8 +5,8 @@
 ;; Author: Jethro Kuan <jethrokuan95@gmail.com>
 ;; URL: https://github.com/jethrokuan/org-roam
 ;; Keywords: org-mode, roam, convenience
-;; Version: 1.1.1
+;; Version: 1.2.1
-;; Package-Requires: ((emacs "26.1") (dash "2.13") (f "0.17.2") (s "1.12.0") (org "9.3") (emacsql "3.0.0") (emacsql-sqlite "1.0.0"))
+;; Package-Requires: ((emacs "26.1") (dash "2.13") (f "0.17.2") (s "1.12.0") (org "9.3") (emacsql "3.0.0") (emacsql-sqlite3 "1.0.0"))
 
 ;; This file is NOT part of GNU Emacs.
 
@@ -44,15 +44,20 @@
 ;; Library Requires
 (require 'cl-lib)
 (require 'org)
-(require 'org-roam-macs)
 (require 'org-element)
+(require 's)
+(require 'dash)
+(require 'org-roam-macs)
 
 (declare-function org-roam-insert "org-roam")
 (declare-function org-roam--get-roam-buffers "org-roam")
 (declare-function org-roam--list-all-files "org-roam")
 (declare-function org-roam--org-roam-file-p "org-roam")
+(declare-function org-roam--str-to-list "org-roam")
+(declare-function org-roam-mode "org-roam")
 
 (defvar org-roam-verbose)
+(defvar org-roam-mode)
 
 (cl-defstruct (org-roam-doctor-checker (:copier nil))
   (name 'missing-checker-name)
@@ -66,7 +71,78 @@
     :description "Fix broken links."
     :actions '(("d" . ("Unlink" . org-roam-doctor--remove-link))
                ("r" . ("Replace link" . org-roam-doctor--replace-link))
-               ("R" . ("Replace link (keep label)" . org-roam-doctor--replace-link-keep-label))))))
+               ("R" . ("Replace link (keep label)" . org-roam-doctor--replace-link-keep-label))))
+   (make-org-roam-doctor-checker
+    :name 'org-roam-doctor-check-roam-props
+    :description "Check #+roam_* properties.")
+   (make-org-roam-doctor-checker
+    :name 'org-roam-doctor-check-tags
+    :description "Check #+roam_tags.")
+   (make-org-roam-doctor-checker
+    :name 'org-roam-doctor-check-alias
+    :description "Check #+roam_alias.")))
+
+(defconst org-roam-doctor--supported-roam-properties
+  '("roam_tags" "roam_alias" "roam_key")
+  "List of supported Org-roam properties.")
+
+(defun org-roam-doctor-check-roam-props (ast)
+  "Checker for detecting invalid #+roam_* properties.
+AST is the org-element parse tree."
+  (let (reports)
+    (org-element-map ast 'keyword
+      (lambda (kw)
+        (let ((key (org-element-property :key kw)))
+          (when (and (string-prefix-p "ROAM_" key t)
+                     (not (member (downcase key) org-roam-doctor--supported-roam-properties)))
+            (push
+             `(,(org-element-property :begin kw)
+               ,(concat "Possible mispelled key: "
+                        (prin1-to-string key)
+                        "\nOrg-roam supports the following keys: "
+                        (s-join ", " org-roam-doctor--supported-roam-properties)))
+             reports)))))
+    reports))
+
+(defun org-roam-doctor-check-tags (ast)
+  "Checker for detecting invalid #+roam_tags.
+AST is the org-element parse tree."
+  (let (reports)
+    (org-element-map ast 'keyword
+      (lambda (kw)
+        (when (string-collate-equalp (org-element-property :key kw) "roam_tags" nil t)
+          (let ((tags (org-element-property :value kw)))
+            (condition-case nil
+                (org-roam--str-to-list tags)
+              (error
+               (push
+                `(,(org-element-property :begin kw)
+                  ,(concat "Unable to parse tags: "
+                           tags
+                           (when (s-contains? "," tags)
+                             "\nCheck that your tags are not comma-separated.")))
+                reports)))))))
+    reports))
+
+(defun org-roam-doctor-check-alias (ast)
+  "Checker for detecting invalid #+roam_alias.
+AST is the org-element parse tree."
+  (let (reports)
+    (org-element-map ast 'keyword
+      (lambda (kw)
+        (when (string-collate-equalp (org-element-property :key kw) "roam_alias" nil t)
+          (let ((aliases (org-element-property :value kw)))
+            (condition-case nil
+              (org-roam--str-to-list aliases)
+              (error
+               (push
+                `(,(org-element-property :begin kw)
+                  ,(concat "Unable to parse aliases: "
+                           aliases
+                           (when (s-contains? "," aliases)
+                             "\nCheck that your aliases are not comma-separated.")))
+                reports)))))))
+    reports))
 
 (defun org-roam-doctor-broken-links (ast)
   "Checker for detecting broken links.
@@ -207,6 +283,7 @@ CHECKER is a org-roam-doctor checker instance."
   "Perform a check on the current buffer to ensure cleanliness.
 If CHECKALL, run the check for all Org-roam files."
   (interactive "P")
+  (unless org-roam-mode (org-roam-mode))
   (let ((files (if checkall
                   (org-roam--list-all-files)
                 (unless (org-roam--org-roam-file-p)

org-roam-faces.el

@@ -0,0 +1,67 @@
+;;; org-roam-faces.el --- Face definitions -*- coding: utf-8; lexical-binding: t; -*-
+
+;; Copyright © 2020 Jethro Kuan <jethrokuan95@gmail.com>
+
+;; Author: Jethro Kuan <jethrokuan95@gmail.com>
+;; URL: https://github.com/org-roam/org-roam
+;; Keywords: org-mode, roam, convenience
+;; Version: 1.2.1
+;; Package-Requires: ((emacs "26.1"))
+
+;; This file is NOT part of GNU Emacs.
+
+;; This program is free software; you can redistribute it and/or modify
+;; it under the terms of the GNU General Public License as published by
+;; the Free Software Foundation; either version 3, or (at your option)
+;; any later version.
+;;
+;; This program is distributed in the hope that it will be useful,
+;; but WITHOUT ANY WARRANTY; without even the implied warranty of
+;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;; GNU General Public License for more details.
+;;
+;; You should have received a copy of the GNU General Public License
+;; along with GNU Emacs; see the file COPYING.  If not, write to the
+;; Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor,
+;; Boston, MA 02110-1301, USA.
+
+;;; Commentary:
+
+;; This file contains the face definitions for Org-roam.
+
+;;; Code:
+
+(defgroup org-roam-faces nil
+  "Faces used by Org-roam."
+  :group 'org-roam
+  :group 'faces)
+
+;;; Definitions
+(defface org-roam-link
+  '((t :inherit org-link))
+  "Face for Org-roam links."
+  :group 'org-roam-faces)
+
+(defface org-roam-link-current
+  '((t :inherit org-link))
+  "Face for Org-roam links pointing to the current buffer."
+  :group 'org-roam-faces)
+
+(defface org-roam-link-invalid
+  '((t :inherit (error org-link)))
+  "Face for Org-roam links that are not valid.
+This face is used for links without a destination."
+  :group 'org-roam-faces)
+
+(defface org-roam-link-shielded
+  '((t :inherit (warning org-link)))
+  "Face for Org-roam links that are shielded.
+This face is used on the region target by `org-roam-insertion'
+during an `org-roam-capture'."
+  :group 'org-roam-faces)
+
+;;; _
+
+(provide 'org-roam-faces)
+
+;;; org-roam-faces.el ends here

org-roam-graph.el

@@ -5,8 +5,8 @@
 ;; Author: Jethro Kuan <jethrokuan95@gmail.com>
 ;; URL: https://github.com/org-roam/org-roam
 ;; Keywords: org-mode, roam, convenience
-;; Version: 1.1.1
+;; Version: 1.2.1
-;; Package-Requires: ((emacs "26.1") (dash "2.13") (f "0.17.2") (s "1.12.0") (org "9.3") (emacsql "3.0.0") (emacsql-sqlite "1.0.0"))
+;; Package-Requires: ((emacs "26.1") (dash "2.13") (f "0.17.2") (s "1.12.0") (org "9.3") (emacsql "3.0.0") (emacsql-sqlite3 "1.0.0"))
 
 ;; This file is NOT part of GNU Emacs.
 
@@ -37,8 +37,10 @@
 
 ;;;; Declarations
 (defvar org-roam-directory)
+(defvar org-roam-mode)
 (declare-function org-roam--org-roam-file-p  "org-roam")
 (declare-function org-roam--path-to-slug     "org-roam")
+(declare-function org-roam-mode              "org-roam")
 
 ;;;; Options
 (defcustom org-roam-graph-viewer (executable-find "firefox")
@@ -163,7 +165,7 @@ The Org-roam database titles table is read, to obtain the list of titles.
 The links table is then read to obtain all directed links, and formatted
 into a digraph."
   (org-roam-db--ensure-built)
-  (org-roam--with-temp-buffer
+  (org-roam--with-temp-buffer nil
     (let* ((nodes (org-roam-db-query node-query))
            (edges-query
             `[:with selected :as [:select [file] :from ,node-query]
@@ -190,14 +192,16 @@ into a digraph."
                                    ","))))
       (dolist (node nodes)
         (let* ((file (xml-escape-string (car node)))
-               (title (or (caadr node)
+               (title (or (cadr node)
                           (org-roam--path-to-slug file)))
                (shortened-title (pcase org-roam-graph-shorten-titles
                                   (`truncate (s-truncate org-roam-graph-max-title-length title))
                                   (`wrap (s-word-wrap org-roam-graph-max-title-length title))
                                   (_ title)))
+               (shortened-title (org-roam-string-quote shortened-title))
+               (title (org-roam-string-quote title))
                (node-properties
-                `(("label"   . ,(s-replace "\"" "\\\"" shortened-title))
+                `(("label"   . ,shortened-title)
                   ("URL"     . ,(concat "org-protocol://roam-file?file=" (url-hexify-string file)))
                   ("tooltip" . ,(xml-escape-string title)))))
           (insert
@@ -217,24 +221,32 @@ into a digraph."
       (insert "}")
       (buffer-string))))
 
-(defun org-roam-graph--build (&optional node-query)
+(defun org-roam-graph--build (&optional node-query callback)
-  "Generate a graph showing the relations between nodes in NODE-QUERY."
+  "Generate a graph showing the relations between nodes in NODE-QUERY.
-  (let ((name org-roam-graph-executable))
+Execute CALLBACK when process exits successfully.
-    (unless (stringp name)
+CALLBACK is passed the graph file as its sole argument."
-      (user-error "`org-roam-graph-executable' is not a string"))
+  (unless (stringp org-roam-graph-executable)
-    (unless (executable-find org-roam-graph-executable)
+    (user-error "`org-roam-graph-executable' is not a string"))
-      (user-error (concat "Cannot find executable \"%s\" to generate the graph.  "
+  (unless (executable-find org-roam-graph-executable)
-                          "Please adjust `org-roam-graph-executable'")
+    (user-error (concat "Cannot find executable \"%s\" to generate the graph.  "
-                  name))
+                        "Please adjust `org-roam-graph-executable'")
-    (let* ((node-query (or node-query
+                org-roam-graph-executable))
-                           `[:select [file titles]
+  (let* ((node-query (or node-query
-                             :from titles
+                         `[:select [file title] :from titles
-                             ,@(org-roam-graph--expand-matcher 'file t)]))
+                           ,@(org-roam-graph--expand-matcher 'file t)
-           (graph (org-roam-graph--dot node-query))
+                           :group :by file]))
-           (temp-dot (make-temp-file "graph." nil ".dot" graph))
+         (graph      (org-roam-graph--dot node-query))
-           (temp-graph (make-temp-file "graph." nil ".svg")))
+         (temp-dot   (make-temp-file "graph." nil ".dot" graph))
-      (call-process name nil 0 nil temp-dot "-Tsvg" "-o" temp-graph)
+         (temp-graph (make-temp-file "graph." nil ".svg")))
-      temp-graph)))
+    (org-roam-message "building graph")
+    (make-process
+     :name "*org-roam-graph--build-process*"
+     :buffer "*org-roam-graph--build-process*"
+     :command `(,org-roam-graph-executable ,temp-dot "-Tsvg" "-o" ,temp-graph)
+     :sentinel (when callback
+                 (lambda (process _event)
+                   (when (= 0 (process-exit-status process))
+                     (funcall callback temp-graph)))))))
 
 (defun org-roam-graph--open (file)
   "Open FILE using `org-roam-graph-viewer' with `view-file' as a fallback."
@@ -249,18 +261,19 @@ into a digraph."
     ('nil (view-file file))
     (_ (signal 'wrong-type-argument `((functionp stringp null) ,org-roam-graph-viewer)))))
 
-(defun org-roam-graph--build-connected-component (file &optional max-distance)
+(defun org-roam-graph--build-connected-component (file &optional max-distance callback)
   "Build a graph of nodes connected to FILE.
-If MAX-DISTANCE is non-nil, limit nodes to MAX-DISTANCE steps."
+If MAX-DISTANCE is non-nil, limit nodes to MAX-DISTANCE steps.
+CALLBACK is passed to `org-roam-graph--build'."
   (let* ((file (file-truename file))
          (files (or (if (and max-distance (>= max-distance 0))
                         (org-roam-db--links-with-max-distance file max-distance)
                       (org-roam-db--connected-component file))
                     (list file)))
-         (query `[:select [file titles]
+         (query `[:select [file title]
                   :from titles
                   :where (in file [,@files])]))
-    (org-roam-graph--build query)))
+    (org-roam-graph--build query callback)))
 
 ;;;; Commands
 ;;;###autoload
@@ -275,6 +288,7 @@ ARG may be any of the following values:
   - `\\[universal-argument]' -   build the graph for FILE.
   - `\\[universal-argument]' -N  build the graph for FILE limiting nodes to N steps."
   (interactive "P")
+  (unless org-roam-mode (org-roam-mode))
   (let ((file (or file (buffer-file-name (buffer-base-buffer)))))
     (unless (or (not arg) (equal arg '(16)))
       (unless file
@@ -282,11 +296,9 @@ ARG may be any of the following values:
       (unless (org-roam--org-roam-file-p file)
         (user-error "\"%s\" is not an org-roam file" file)))
     (pcase arg
-      ('nil            (org-roam-graph--open (org-roam-graph--build node-query)))
+      ('nil            (org-roam-graph--build node-query #'org-roam-graph--open))
-      ('(4)            (org-roam-graph--open (org-roam-graph--build-connected-component file)))
+      ('(4)            (org-roam-graph--build-connected-component file nil #'org-roam-graph--open))
-      ((pred integerp) (let ((graph (org-roam-graph--build-connected-component file (abs arg))))
+      ((pred integerp) (org-roam-graph--build-connected-component file (abs arg) (when (>= arg 0) #'org-roam-graph--open)))
-                         (when (>= arg 0)
-                           (org-roam-graph--open graph))))
       ('(16)           (org-roam-graph--build node-query))
       ('-              (org-roam-graph--build-connected-component file))
       (_ (user-error "Unrecognized ARG: %s" arg)))))

org-roam-macs.el

@@ -5,8 +5,8 @@
 ;; Author: Jethro Kuan <jethrokuan95@gmail.com>
 ;; URL: https://github.com/org-roam/org-roam
 ;; Keywords: org-mode, roam, convenience
-;; Version: 1.1.1
+;; Version: 1.2.1
-;; Package-Requires: ((emacs "26.1") (dash "2.13") (f "0.17.2") (s "1.12.0") (org "9.3") (emacsql "3.0.0") (emacsql-sqlite "1.0.0"))
+;; Package-Requires: ((emacs "26.1") (dash "2.13") (f "0.17.2") (s "1.12.0") (org "9.3") (emacsql "3.0.0") (emacsql-sqlite3 "1.0.0"))
 
 ;; This file is NOT part of GNU Emacs.
 
@@ -33,17 +33,24 @@
 ;;
 ;;; Code:
 ;;;; Library Requires
+(require 'dash)
 
 (defvar org-roam-verbose)
 
-(defmacro org-roam--with-temp-buffer (&rest body)
+(defmacro org-roam--with-temp-buffer (file &rest body)
   "Execute BODY within a temp buffer.
-Like `with-temp-buffer', but propagates `org-roam-directory'."
+Like `with-temp-buffer', but propagates `org-roam-directory'.
-  (declare (indent 0) (debug t))
+If FILE, set `org-roam-temp-file-name' to file and insert its contents."
+  (declare (indent 1) (debug t))
   (let ((current-org-roam-directory (make-symbol "current-org-roam-directory")))
     `(let ((,current-org-roam-directory org-roam-directory))
        (with-temp-buffer
-         (let ((org-roam-directory ,current-org-roam-directory))
+         (let ((org-roam-directory ,current-org-roam-directory)
+               (org-mode-hook nil))
+           (org-mode)
+           (when ,file
+             (insert-file-contents ,file)
+             (setq-local org-roam-file-name ,file))
            ,@body)))))
 
 (defmacro org-roam--with-template-error (templates &rest body)
@@ -64,6 +71,34 @@ to look.
   (when org-roam-verbose
     (apply #'message `(,(concat "(org-roam) " format-string) ,@args))))
 
+(defun org-roam-string-quote (str)
+  "Quote STR."
+  (->> str
+       (s-replace "\\" "\\\\")
+       (s-replace "\"" "\\\"")))
+
+;;; Shielding regions
+(defun org-roam-shield-region (beg end)
+  "Shield REGION against modifications.
+REGION must be a cons-cell containing the marker to the region
+beginning and maximum values."
+  (when (and beg end)
+    (add-text-properties beg end
+                           '(font-lock-face org-roam-link-shielded
+                                            read-only t)
+                           (marker-buffer beg))
+    (cons beg end)))
+
+(defun org-roam-unshield-region (beg end)
+  "Unshield the shielded REGION."
+  (when (and beg end)
+    (let ((inhibit-read-only t))
+      (remove-text-properties beg end
+                              '(font-lock-face org-roam-link-shielded
+                                               read-only t)
+                              (marker-buffer beg)))
+    (cons beg end)))
+
 (provide 'org-roam-macs)
 
 ;;; org-roam-macs.el ends here

org-roam-protocol.el

@@ -4,7 +4,7 @@
 ;; Author: Jethro Kuan <jethrokuan95@gmail.com>
 ;; URL: https://github.com/org-roam/org-roam
 ;; Keywords: org-mode, roam, convenience
-;; Version: 1.1.1
+;; Version: 1.2.1
 ;; Package-Requires: ((emacs "26.1") (org "9.3"))
 
 ;; This file is NOT part of GNU Emacs.
@@ -56,7 +56,7 @@ It opens or creates a note with the given ref.
     (unless (assoc 'ref decoded-alist)
       (error "No ref key provided"))
     (when-let ((title (cdr (assoc 'title decoded-alist))))
-      (push (cons 'slug (org-roam--title-to-slug title)) decoded-alist))
+      (push (cons 'slug (funcall org-roam-title-to-slug-function title)) decoded-alist))
     (let* ((org-roam-capture-templates org-roam-capture-ref-templates)
            (org-roam-capture--context 'ref)
            (org-roam-capture--info decoded-alist)
@@ -78,7 +78,7 @@ It should contain the FILE key, pointing to the path of the file to open.
 org-protocol://roam-file?file=/path/to/file.org"
   (when-let ((file (plist-get info :file)))
     (raise-frame)
-    (find-file file))
+    (org-roam--find-file file))
   nil)
 
 (push '("org-roam-ref"  :protocol "roam-ref"   :function org-roam-protocol-open-ref)

tests/roam-files/alias.org

@@ -1,2 +1,2 @@
-#+ROAM_ALIAS: "a1" "a 2"
+#+roam_alias: "a1" "a 2"
-#+TITLE: t1
+#+title: t1

tests/roam-files/bar.org

@@ -1,3 +1,3 @@
-#+TITLE: Bar
+#+title: Bar
 
 This is file bar. Bar links to [[file:nested/bar.org][Nested Bar]].

tests/roam-files/base.org

@@ -1 +1 @@
-#+TITLE: Base
+#+title: Base

tests/roam-files/foo.org

@@ -1,4 +1,4 @@
-#+TITLE: Foo
+#+title: Foo
 
 This is the foo file. It contains a link to [[file:bar.org][Bar]].
 

tests/roam-files/headlines/headline.org

@@ -0,0 +1,14 @@
+#+TITLE: Headline
+
+* Headline 1
+:PROPERTIES:
+:ID:       e84d0630-efad-4017-9059-5ef917908823
+:END:
+
+* No headline here
+Oops.
+
+* Headline 2
+:PROPERTIES:
+:ID:       801b58eb-97e2-435f-a33e-ff59a2f0c213
+:END:

tests/roam-files/nested/bar.org

@@ -1,3 +1,3 @@
-#+TITLE: Nested Bar
+#+title: Nested Bar
 
 This file is nested, 1 level deeper. It links to both [[file:../foo.org][Foo]] and [[file:foo.org][Nested Foo]].

tests/roam-files/nested/deeply/deeply_nested_file.org

@@ -1 +1 @@
-#+TITLE: Deeply Nested File
+#+title: Deeply Nested File

tests/roam-files/nested/foo.org

@@ -1,3 +1,3 @@
-#+TITLE: Nested Foo
+#+title: Nested Foo
 
 This file has no links.

tests/roam-files/tags/no_tag.org

@@ -1,3 +1,3 @@
-#+TITLE: Tagless File
+#+title: Tagless File
 
-This file has no tags, and should not yield any tags on extracting via =#+ROAM_TAGS=.
+This file has no tags, and should not yield any tags on extracting via ~#+roam_tags~.

tests/roam-files/tags/tag.org

@@ -1,4 +1,4 @@
-#+ROAM_TAGS: "t1" "t2 with space" t3
+#+roam_tags: "t1" "t2 with space" t3
-#+TITLE: Tags
+#+title: Tags
 
 This file is used to test functionality for =(org-roam--extract-tags)=

tests/roam-files/titles/aliases.org

@@ -1 +1 @@
-#+ROAM_ALIAS: "roam" "alias"
+#+roam_alias: "roam" "alias"

tests/roam-files/titles/combination.org

@@ -1,4 +1,4 @@
-#+TITLE: TITLE PROP
+#+title: TITLE PROP
-#+ROAM_ALIAS: "roam" "alias"
+#+roam_alias: "roam" "alias"
 
 * Headline

tests/roam-files/titles/title.org

@@ -1 +1 @@
-#+TITLE: Title
+#+title: Title

tests/roam-files/unlinked.org

@@ -1,3 +1,3 @@
-#+TITLE: Unlinked
+#+title: Unlinked
 
 Nothing links here :(

tests/roam-files/web_ref.org

@@ -1 +1 @@
-#+ROAM_KEY: https://google.com/
+#+roam_key: https://google.com/

tests/test-org-roam-perf.el

@@ -0,0 +1,53 @@
+;;; test-org-roam-perf.el --- Performance Tests for Org-roam -*- lexical-binding: t; -*-
+
+;; Copyright (C) 2020 Jethro Kuan
+
+;; Author: Jethro Kuan <jethrokuan95@gmail.com>
+;; Package-Requires: ((buttercup))
+
+;; This program is free software; you can redistribute it and/or modify
+;; it under the terms of the GNU General Public License as published by
+;; the Free Software Foundation, either version 3 of the License, or
+;; (at your option) any later version.
+
+;; This program is distributed in the hope that it will be useful,
+;; but WITHOUT ANY WARRANTY; without even the implied warranty of
+;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;; GNU General Public License for more details.
+
+;; You should have received a copy of the GNU General Public License
+;; along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+;;; Commentary:
+;;; Code:
+
+(require 'buttercup)
+(require 'org-roam)
+
+(defconst test-org-roam-perf-zip-url "https://github.com/org-roam/test-org-files/archive/master.zip"
+  "Path to zip for test org-roam files.")
+
+(defun test-org-roam-perf--abs-path (file-path)
+  "Get absolute FILE-PATH from `org-roam-directory'."
+  (file-truename (expand-file-name file-path org-roam-directory)))
+
+(defun test-org-roam-perf--init ()
+  "."
+  (let* ((temp-loc (expand-file-name (make-temp-name "test-org-files-") temporary-file-directory))
+         (zip-file-loc (concat temp-loc ".zip"))
+         (_ (url-copy-file test-org-roam-perf-zip-url zip-file-loc))
+         (_ (shell-command (format "mkdir -p %s && unzip -j -qq %s -d %s" temp-loc zip-file-loc temp-loc))))
+    (setq org-roam-directory temp-loc)))
+
+(describe "Cache Build"
+  (it "cache build from scratch time to be acceptable"
+    (test-org-roam-perf--init)
+    (pcase (benchmark-run 1 (org-roam-db-build-cache t))
+      (`(,time ,gcs ,time-in-gc)
+       (message "Elapsed time: %fs (%fs in %d GCs)" time time-in-gc gcs)
+       (expect time :to-be-less-than 100))))
+  (it "builds quickly without change"
+    (pcase (benchmark-run 1 (org-roam-db-build-cache))
+      (`(,time ,gcs ,time-in-gc)
+       (message "Elapsed time: %fs (%fs in %d GCs)" time time-in-gc gcs)
+       (expect time :to-be-less-than 5)))))

tests/test-org-roam.el

@@ -3,7 +3,7 @@
 ;; Copyright (C) 2020 Jethro Kuan
 
 ;; Author: Jethro Kuan <jethrokuan95@gmail.com>
-;; Package-Requires: ((buttercup) (with-simulated-input))
+;; Package-Requires: ((buttercup))
 
 ;; This program is free software; you can redistribute it and/or modify
 ;; it under the terms of the GNU General Public License as published by
@@ -22,7 +22,6 @@
 ;;; Code:
 
 (require 'buttercup)
-(require 'with-simulated-input)
 (require 'org-roam)
 (require 'dash)
 
@@ -53,6 +52,25 @@
   (delete-file (org-roam-db--get))
   (org-roam-db--close))
 
+(describe "org-roam--str-to-list"
+  (it "nil"
+    (expect (org-roam--str-to-list nil)
+            :to-be
+            nil))
+  (it "\"multi word\" prop 123"
+    (expect (org-roam--str-to-list "\"multi word\" prop 123")
+            :to-equal
+            '("multi word" "prop" "123")))
+  (it "prop \"multi word\" 123"
+    (expect (org-roam--str-to-list "\"multi word\" prop 123")
+            :to-equal
+            '("multi word" "prop" "123")))
+  (it "errors on bad input"
+    (expect (org-roam--str-to-list 1)
+            :to-throw)
+    (expect (org-roam--str-to-list "\"hello")
+            :to-throw)))
+
 (describe "Title extraction"
   :var (org-roam-title-sources)
   (before-all
@@ -193,6 +211,20 @@
               :to-equal
               '("deeply")))
 
+    (it "extracts from first directory"
+      (expect (test #'org-roam--extract-tags-first-directory
+                    "base.org")
+              :to-equal
+              nil)
+      (expect (test #'org-roam--extract-tags-first-directory
+                    "tags/tag.org")
+              :to-equal
+              '("tags"))
+      (expect (test #'org-roam--extract-tags-first-directory
+                    "nested/deeply/deeply_nested_file.org")
+              :to-equal
+              '("nested")))
+
     (describe "uses org-roam-tag-sources correctly"
       (it "'(prop)"
         (expect (let ((org-roam-tag-sources '(prop)))
@@ -207,6 +239,26 @@
                 :to-equal
                 '("t1" "t2 with space" "t3" "tags"))))))
 
+(describe "Headline extraction"
+  (before-all
+    (test-org-roam--init))
+
+  (after-all
+    (test-org-roam--teardown))
+
+  (cl-flet
+      ((test (fn file)
+             (let* ((fname (test-org-roam--abs-path file))
+                    (buf (find-file-noselect fname)))
+               (with-current-buffer buf
+                 (funcall fn fname)))))
+    (it "extracts headlines"
+      (expect (test #'org-roam--extract-headlines
+                    "headlines/headline.org")
+              :to-equal
+              `(["e84d0630-efad-4017-9059-5ef917908823" ,(test-org-roam--abs-path "headlines/headline.org")]
+                ["801b58eb-97e2-435f-a33e-ff59a2f0c213" ,(test-org-roam--abs-path "headlines/headline.org")])))))
+
 ;;; Tests
 (xdescribe "org-roam-db-build-cache"
   (before-each
@@ -270,199 +322,6 @@
             :to-equal
             (list :files 0 :links 0 :tags 0 :titles 0 :refs 0 :deleted 0))))
 
-(xdescribe "org-roam-insert"
-  (before-each
-    (test-org-roam--init))
-
-  (after-each
-    (test-org-roam--teardown))
-
-  (it "temp1 -> foo"
-    (let ((buf (test-org-roam--find-file "temp1.org")))
-      (with-current-buffer buf
-        (with-simulated-input
-         "Foo RET"
-         (org-roam-insert))))
-    (expect (buffer-string) :to-match (regexp-quote "file:foo.org")))
-
-  (it "temp2 -> nested/foo"
-    (let ((buf (test-org-roam--find-file "temp2.org")))
-      (with-current-buffer buf
-        (with-simulated-input
-         "(nested) SPC Nested SPC Foo RET"
-         (org-roam-insert))))
-    (expect (buffer-string) :to-match (regexp-quote "file:nested/foo.org")))
-
-  (it "nested/temp3 -> foo"
-    (let ((buf (test-org-roam--find-file "nested/temp3.org")))
-      (with-current-buffer buf
-        (with-simulated-input
-         "Foo RET"
-         (org-roam-insert))))
-    (expect (buffer-string) :to-match (regexp-quote "file:../foo.org")))
-
-  (it "a/b/temp4 -> nested/foo"
-    (let ((buf (test-org-roam--find-file "a/b/temp4.org")))
-      (with-current-buffer buf
-        (with-simulated-input
-         "(nested) SPC Nested SPC Foo RET"
-         (org-roam-insert))))
-    (expect (buffer-string) :to-match (regexp-quote "file:../../nested/foo.org"))))
-
-(xdescribe "rename file updates cache"
-  (before-each
-    (test-org-roam--init))
-
-  (after-each
-    (test-org-roam--teardown))
-
-  (it "foo -> new_foo"
-    (rename-file (test-org-roam--abs-path "foo.org")
-                 (test-org-roam--abs-path "new_foo.org"))
-    ;; Cache should be cleared of old file
-    (expect (caar (org-roam-db-query [:select (funcall count)
-                                      :from titles
-                                      :where (= file $s1)]
-                                     (test-org-roam--abs-path "foo.org"))) :to-be 0)
-    (expect (caar (org-roam-db-query [:select (funcall count)
-                                      :from refs
-                                      :where (= file $s1)]
-                                     (test-org-roam--abs-path "foo.org"))) :to-be 0)
-    (expect (caar (org-roam-db-query [:select (funcall count)
-                                      :from links
-                                      :where (= from $s1)]
-                                     (test-org-roam--abs-path "foo.org"))) :to-be 0)
-
-    ;; Cache should be updated
-    (expect (org-roam-db-query [:select [to]
-                                :from links
-                                :where (= from $s1)]
-                               (test-org-roam--abs-path "new_foo.org"))
-            :to-have-same-items-as
-            (list (list (test-org-roam--abs-path "bar.org"))))
-    (expect (org-roam-db-query [:select [from]
-                                :from links
-                                :where (= to $s1)]
-                               (test-org-roam--abs-path "new_foo.org"))
-            :to-have-same-items-as
-            (list (list (test-org-roam--abs-path "nested/bar.org"))))
-
-    ;; Links are updated
-    (expect (with-temp-buffer
-              (insert-file-contents (test-org-roam--abs-path "nested/bar.org"))
-              (buffer-string))
-            :to-match
-            (regexp-quote "[[file:../new_foo.org][Foo]]")))
-
-  (it "foo -> foo with spaces"
-    (rename-file (test-org-roam--abs-path "foo.org")
-                 (test-org-roam--abs-path "foo with spaces.org"))
-    ;; Cache should be cleared of old file
-    (expect (caar (org-roam-db-query [:select (funcall count)
-                                      :from titles
-                                      :where (= file $s1)]
-                                     (test-org-roam--abs-path "foo.org"))) :to-be 0)
-    (expect (caar (org-roam-db-query [:select (funcall count)
-                                      :from refs
-                                      :where (= file $s1)]
-                                     (test-org-roam--abs-path "foo.org"))) :to-be 0)
-    (expect (caar (org-roam-db-query [:select (funcall count)
-                                      :from links
-                                      :where (= from $s1)]
-                                     (test-org-roam--abs-path "foo.org"))) :to-be 0)
-
-    ;; Cache should be updated
-    (expect (org-roam-db-query [:select [to]
-                                :from links
-                                :where (= from $s1)]
-                               (test-org-roam--abs-path "foo with spaces.org"))
-            :to-have-same-items-as
-            (list (list (test-org-roam--abs-path "bar.org"))))
-    (expect (org-roam-db-query [:select [from]
-                                :from links
-                                :where (= to $s1)]
-                               (test-org-roam--abs-path "foo with spaces.org"))
-            :to-have-same-items-as
-            (list (list (test-org-roam--abs-path "nested/bar.org"))))
-
-    ;; Links are updated
-    (expect (with-temp-buffer
-              (insert-file-contents (test-org-roam--abs-path "nested/bar.org"))
-              (buffer-string))
-            :to-match
-            (regexp-quote "[[file:../foo with spaces.org][Foo]]")))
-
-  (it "no-title -> meaningful-title"
-    (rename-file (test-org-roam--abs-path "no-title.org")
-                 (test-org-roam--abs-path "meaningful-title.org"))
-    ;; File has no forward links
-    (expect (caar (org-roam-db-query [:select (funcall count)
-                                      :from links
-                                      :where (= from $s1)]
-                                     (test-org-roam--abs-path "no-title.org"))) :to-be 0)
-    (expect (caar (org-roam-db-query [:select (funcall count)
-                                      :from links
-                                      :where (= from $s1)]
-                                     (test-org-roam--abs-path "meaningful-title.org"))) :to-be 1)
-
-    ;; Links are updated with the appropriate name
-    (expect (with-temp-buffer
-              (insert-file-contents (test-org-roam--abs-path "meaningful-title.org"))
-              (buffer-string))
-            :to-match
-            (regexp-quote "[[file:meaningful-title.org][meaningful-title]]")))
-
-  (it "web_ref -> hello"
-    (expect (org-roam-db-query
-             [:select [file] :from refs
-              :where (= ref $s1)]
-             "https://google.com/")
-            :to-equal
-            (list (list (test-org-roam--abs-path "web_ref.org"))))
-    (rename-file (test-org-roam--abs-path "web_ref.org")
-                 (test-org-roam--abs-path "hello.org"))
-    (expect (org-roam-db-query
-             [:select [file] :from refs
-              :where (= ref $s1)]
-             "https://google.com/")
-            :to-equal (list (list (test-org-roam--abs-path "hello.org"))))
-    (expect (caar (org-roam-db-query
-                   [:select [ref] :from refs
-                    :where (= file $s1)]
-                   (test-org-roam--abs-path "web_ref.org")))
-            :to-equal nil)))
-
-(xdescribe "delete file updates cache"
-  (before-each
-    (test-org-roam--init))
-
-  (after-each
-    (test-org-roam--teardown))
-
-  (it "delete foo"
-    (delete-file (test-org-roam--abs-path "foo.org"))
-    (expect (caar (org-roam-db-query [:select (funcall count)
-                                      :from titles
-                                      :where (= file $s1)]
-                                     (test-org-roam--abs-path "foo.org"))) :to-be 0)
-    (expect (caar (org-roam-db-query [:select (funcall count)
-                                      :from refs
-                                      :where (= file $s1)]
-                                     (test-org-roam--abs-path "foo.org"))) :to-be 0)
-    (expect (caar (org-roam-db-query [:select (funcall count)
-                                      :from links
-                                      :where (= from $s1)]
-                                     (test-org-roam--abs-path "foo.org"))) :to-be 0))
-
-  (it "delete web_ref"
-    (expect (org-roam-db-query [:select * :from refs])
-            :to-have-same-items-as
-            (list (list "https://google.com/" (test-org-roam--abs-path "web_ref.org") "website")))
-    (delete-file (test-org-roam--abs-path "web_ref.org"))
-    (expect (org-roam-db-query [:select * :from refs])
-            :to-have-same-items-as
-            (list))))
-
 (provide 'test-org-roam)
 
 ;;; test-org-roam.el ends here