From 340215a16aa379bdd6121303c729a405cb481bef Mon Sep 17 00:00:00 2001 From: Jethro Kuan Date: Sun, 29 Aug 2021 19:33:14 +0800 Subject: [PATCH] (feat) core: support new org-mode citations (#1806) Support caching the new Org 9.5 citations. Because citations now has first-class support, and are treated differently from links, they are now cached in their own table. Org-ref citations, instead of being stored in the links table, are now stored in the citations table instead. To use a citation as a ROAM_REF, use the `@citeKey` syntax --- CHANGELOG.md | 2 + doc/org-roam.org | 43 +++++++++++++++++ doc/org-roam.texi | 78 +++++++++++++++++++++++++++---- org-roam-db.el | 115 +++++++++++++++++++++++++++------------------- org-roam-mode.el | 29 +++++++----- 5 files changed, 199 insertions(+), 68 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 63b205a..9b380a6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,8 @@ ## TBD ### Added +- [#1806](https://github.com/org-roam/org-roam/pull/1806) db: support caching and usage of Org 9.5's in-built citations + ### Removed ### Changed - [#1795](https://github.com/org-roam/org-roam/pull/1795) buffer: optimized reflinks fetch diff --git a/doc/org-roam.org b/doc/org-roam.org index 33817d6..17c7c25 100644 --- a/doc/org-roam.org +++ b/doc/org-roam.org @@ -646,6 +646,49 @@ Org-roam also provides some functions to add or remove refs. Remove a ref from the node at point. +* Citations + +Since version 9.5, Org has first-class support for citations. Org-roam supports +the caching of both these in-built citations (of form ~[cite:@key]~) and [[https://github.com/jkitchin/org-ref][org-ref]] +citations (of form cite:key). + +Org-roam attempts to load both the ~org-ref~ and ~org-cite~ package when +indexing files, so no further setup from the user is required for citation +support. + +** Using the Cached Information + +It is common to use take reference notes for academic papers. To designate the +node to be the canonical node for the academic paper, we can use its unique +citation key: + +#+begin_src org +,* Probabilistic Robotics +:PROPERTIES: +:ID: 51b7b82c-bbb4-4822-875a-ed548cffda10 +:ROAM_REFS: @thrun2005probabilistic +:END: +#+end_src + +for ~org-cite~, or: + +#+begin_src org +,* Probabilistic Robotics +:PROPERTIES: +:ID: 51b7b82c-bbb4-4822-875a-ed548cffda10 +:ROAM_REFS: cite:thrun2005probabilistic +:END: +#+end_src + +for ~org-ref~. + +When another node has a citation for that key, we can see it using the +~Reflinks~ section of the Org-roam buffer. + +Extension developers may be interested in retrieving the citations within their +notes. This information can be found within the ~citation~ table of the Org-roam +database. + * Completion Completions for Org-roam are provided via ~completion-at-point~. Org-roam diff --git a/doc/org-roam.texi b/doc/org-roam.texi index dc310c8..a07f6ea 100644 --- a/doc/org-roam.texi +++ b/doc/org-roam.texi @@ -70,6 +70,7 @@ General Public License for more details. * Customizing Node Caching:: * The Org-roam Buffer:: * Node Properties:: +* Citations:: * Completion:: * Encryption:: * Org-roam Protocol:: @@ -86,6 +87,7 @@ General Public License for more details. * Command Index:: * Function Index:: * Variable Index:: +* Bibliography: Bibliography (1). @detailmenu --- The Detailed Node Listing --- @@ -126,6 +128,10 @@ Node Properties * Tags:: * Refs:: +Citations + +* Using the Cached Information:: + Completion * Completing within Link Brackets:: @@ -250,7 +256,7 @@ available to Emacs. Org-roam is a tool that will appear unfriendly to anyone unfamiliar with Emacs and Org-mode, but it is also extremely powerful to those willing to put effort -inn mastering the intricacies. Org-roam stands on the shoulders of giants. Emacs +in mastering the intricacies. Org-roam stands on the shoulders of giants. Emacs was first created in 1976, and remains the tool of choice for many for editing text and designing textual interfaces. The malleability of Emacs allowed the creation of Org-mode, an all-purpose plain-text system for maintaining TODO @@ -637,8 +643,8 @@ The @code{file-truename} function is only necessary when you use symbolic links inside @code{org-roam-directory}: Org-roam does not resolve symbolic links. Next, we setup Org-roam to run functions on file changes to maintain cache -consistency. This is achieved by running @code{M-x org-roam-db-autosync-mode~}. -To ensure that Org-roam is available on startup, place this in your Emacs +consistency. This is achieved by running @code{M-x org-roam-db-autosync-mode}. To +ensure that Org-roam is available on startup, place this in your Emacs configuration: @lisp @@ -995,6 +1001,55 @@ ref to add. Remove a ref from the node at point. @end defun +@node Citations +@chapter Citations + +Since version 9.5, Org has first-class support for citations. Org-roam supports +the caching of both these in-built citations (of form @code{[cite:@@key]}) and @uref{https://github.com/jkitchin/org-ref, org-ref} +citations (of form (NO@math{_ITEM}@math{_DATA}:key)). + +Org-roam attempts to load both the @code{org-ref} and @code{org-cite} package when +indexing files, so no further setup from the user is required for citation +support. + +@menu +* Using the Cached Information:: +@end menu + +@node Using the Cached Information +@section Using the Cached Information + +It is common to use take reference notes for academic papers. To designate the +node to be the canonical node for the academic paper, we can use its unique +citation key: + +@example +* Probabilistic Robotics +:PROPERTIES: +:ID: 51b7b82c-bbb4-4822-875a-ed548cffda10 +:ROAM_REFS: @@thrun2005probabilistic +:END: +@end example + +for @code{org-cite}, or: + +@example +* Probabilistic Robotics +:PROPERTIES: +:ID: 51b7b82c-bbb4-4822-875a-ed548cffda10 +:ROAM_REFS: cite:thrun2005probabilistic +:END: +@end example + +for @code{org-ref}. + +When another node has a citation for that key, we can see it using the +@code{Reflinks} section of the Org-roam buffer. + +Extension developers may be interested in retrieving the citations within their +notes. This information can be found within the @code{citation} table of the Org-roam +database. + @node Completion @chapter Completion @@ -1331,12 +1386,12 @@ here. This template means don't insert any content, but place the cursor here. @item -@code{:target} is a compulsory specification in the Org-roam capture -template. The first element of the list indicates the type of the -target, the second element indicates the location of the captured -node, and the rest of the elements indicate prefilled template that -will be inserted and the position of the point will be adjusted for. -The latter behavior various from type to type of the capture target. +@code{:target} is a compulsory specification in the Org-roam capture template. The +first element of the list indicates the type of the target, the second +element indicates the location of the captured node, and the rest of the +elements indicate prefilled template that will be inserted and the position +of the point will be adjusted for. The latter behavior various from type to +type of the capture target. @item @code{:unnarrowed t} tells org-capture to show the contents for the whole file, @@ -2121,5 +2176,10 @@ When GOTO is non-nil, go the note without creating an entry." @printindex vr +@node Bibliography (1) +@chapter Bibliography + +NO@math{_ITEM}@math{_DATA}:key + Emacs 28.0.50 (Org mode 9.5) @bye diff --git a/org-roam-db.el b/org-roam-db.el index fc96376..bbab388 100644 --- a/org-roam-db.el +++ b/org-roam-db.el @@ -79,7 +79,7 @@ slow." :group 'org-roam) ;;; Variables -(defconst org-roam-db-version 16) +(defconst org-roam-db-version 17) ;; TODO Rename this (defconst org-roam--sqlite-available-p @@ -168,6 +168,13 @@ The query is expected to be able to fail, in this situation, run HANDLER." alias] (:foreign-key [node-id] :references nodes [id] :on-delete :cascade))) + (citations + ([(node-id :not-null) + (cite-key :not-null) + (pos :not-null) + properties] + (:foreign-key [node-id] :references nodes [id] :on-delete :cascade))) + (refs ([(node-id :not-null) (ref :not-null) @@ -284,14 +291,23 @@ If UPDATE-P is non-nil, first remove the file in the database." (dolist (fn fns) (funcall fn))))))) -(defun org-roam-db-map-links (fns) - "Run FNS over all links in the current buffer." +(defun org-roam-db-map-links (info fns) + "Run FNS over all links in the current buffer. +INFO is the org-element parsed buffer." (org-with-point-at 1 - (org-element-map (org-element-parse-buffer) 'link + (org-element-map info 'link (lambda (link) (dolist (fn fns) (funcall fn link)))))) +(defun org-roam-db-map-citations (info fns) + "Run FNS over all citations in the current buffer. +INFO is the org-element parsed buffer." + (org-element-map info 'citation-reference + (lambda (cite) + (dolist (fn fns) + (funcall fn cite))))) + (defun org-roam-db-insert-file-node () "Insert the file-level node into the Org-roam cache." (org-with-point-at 1 @@ -309,10 +325,7 @@ If UPDATE-P is non-nil, first remove the file in the database." (scheduled nil) (deadline nil) (level 0) - (aliases (org-entry-get (point) "ROAM_ALIASES")) - (aliases (when aliases (split-string-and-unquote aliases))) (tags org-file-tags) - (refs (org-entry-get (point) "ROAM_REFS")) (properties (org-entry-properties)) (olp nil)) (org-roam-db-query! @@ -331,29 +344,8 @@ If UPDATE-P is non-nil, first remove the file in the database." (mapcar (lambda (tag) (vector id (substring-no-properties tag))) tags))) - (when aliases - (org-roam-db-query - [:insert :into aliases - :values $v1] - (mapcar (lambda (alias) - (vector id alias)) - aliases))) - (when refs - (setq refs (split-string-and-unquote refs)) - (let (rows) - (dolist (ref refs) - (if (string-match org-link-plain-re ref) - (progn - (push (vector id (match-string 2 ref) - (match-string 1 ref)) rows)) - (lwarn '(org-roam) :warning - "%s:%s\tInvalid ref %s, skipping..." - (buffer-file-name) (point) ref))) - (when rows - (org-roam-db-query - [:insert :into refs - :values $v1] - rows))))))))) + (org-roam-db-insert-aliases) + (org-roam-db-insert-refs)))))) (cl-defun org-roam-db-insert-node-data () "Insert node data for headline at point into the Org-roam cache." @@ -413,11 +405,13 @@ If UPDATE-P is non-nil, first remove the file in the database." (let (rows) (dolist (ref refs) (save-match-data - (if (string-match org-link-plain-re ref) - (progn - (push (vector node-id (match-string 2 ref) (match-string 1 ref)) rows)) - (lwarn '(org-roam) :warning - "%s:%s\tInvalid ref %s, skipping..." (buffer-file-name) (point) ref)))) + (cond ((string-equal (substring ref 0 1) "@") + (push (vector node-id (substring ref 1) "cite") rows)) + ((string-match org-link-plain-re ref) + (push (vector node-id (match-string 2 ref) (match-string 1 ref)) rows)) + (t + (lwarn '(org-roam) :warning + "%s:%s\tInvalid ref %s, skipping..." (buffer-file-name) (point) ref))))) (when rows (org-roam-db-query [:insert :into refs :values $v1] @@ -429,24 +423,42 @@ If UPDATE-P is non-nil, first remove the file in the database." (goto-char (org-element-property :begin link)) (let ((type (org-element-property :type link)) (path (org-element-property :path link)) + (source (org-roam-id-at-point)) (properties (list :outline (ignore-errors ;; This can error if link is not under any headline - (org-get-outline-path 'with-self 'use-cache)))) - (source (org-roam-id-at-point))) + (org-get-outline-path 'with-self 'use-cache))))) ;; For Org-ref links, we need to split the path into the cite keys - (when (and (boundp 'org-ref-cite-types) + (when (and source path) + (if (and (require 'org-ref nil 'noerror) + (boundp 'org-ref-cite-types) (fboundp 'org-ref-split-and-strip-string) (member type org-ref-cite-types)) - (setq path (org-ref-split-and-strip-string path))) - (unless (listp path) - (setq path (list path))) - (when (and source path) + (progn + (setq path (org-ref-split-and-strip-string path)) + (org-roam-db-query + [:insert :into citations + :values $v1] + (mapcar (lambda (p) (vector source p (point) properties)) path))) + + (org-roam-db-query + [:insert :into links + :values $v1] + (vector (point) source path type properties))))))) + +(defun org-roam-db-insert-citation (citation) + "Insert data for CITATION at current point into the Org-roam cache." + (save-excursion + (goto-char (org-element-property :begin citation)) + (let ((key (org-element-property :key citation)) + (source (org-roam-id-at-point)) + (properties (list :outline (ignore-errors + ;; This can error if link is not under any headline + (org-get-outline-path 'with-self 'use-cache))))) + (when (and source key) (org-roam-db-query - [:insert :into links + [:insert :into citations :values $v1] - (mapcar (lambda (p) - (vector (point) source p type properties)) - path)))))) + (vector source key (point) properties)))))) ;;;; Fetching (defun org-roam-db--get-current-files () @@ -479,7 +491,8 @@ If the file exists, update the cache with information." (setq file-path (or file-path (buffer-file-name (buffer-base-buffer)))) (let ((content-hash (org-roam-db--file-hash file-path)) (db-hash (caar (org-roam-db-query [:select hash :from files - :where (= file $s1)] file-path)))) + :where (= file $s1)] file-path))) + info) (unless (string= content-hash db-hash) (org-roam-with-file file-path nil (save-excursion @@ -494,8 +507,14 @@ If the file exists, update the cache with information." #'org-roam-db-insert-tags #'org-roam-db-insert-refs)) (setq org-outline-path-cache nil) + (setq info (org-element-parse-buffer)) (org-roam-db-map-links - (list #'org-roam-db-insert-link))))))) + info + (list #'org-roam-db-insert-link)) + (when (require 'org-cite nil 'noerror) + (org-roam-db-map-citations + info + (list #'org-roam-db-insert-citation)))))))) ;;;###autoload (defun org-roam-db-sync (&optional force) diff --git a/org-roam-mode.el b/org-roam-mode.el index f10331f..1c79b9b 100644 --- a/org-roam-mode.el +++ b/org-roam-mode.el @@ -546,7 +546,14 @@ Sorts by title." :from refs :left-join links :where (= refs:node-id $s1) - :and (= links:dest refs:ref)] + :and (= links:dest refs:ref) + :union + :select :distinct [refs:ref citations:node-id + citations:pos citations:properties] + :from refs + :left-join citations + :where (= refs:node-id $s1) + :and (= citations:cite-key refs:ref)] (org-roam-node-id node))) links) (pcase-dolist (`(,ref ,source-id ,pos ,properties) refs) @@ -566,16 +573,16 @@ Sorts by title." (defun org-roam-reflinks-section (node) "The reflinks section for NODE." - (when (org-roam-node-refs node) - (let* ((reflinks (seq-sort #'org-roam-reflinks-sort (org-roam-reflinks-get node)))) - (magit-insert-section (org-roam-reflinks) - (magit-insert-heading "Reflinks:") - (dolist (reflink reflinks) - (org-roam-node-insert-section - :source-node (org-roam-reflink-source-node reflink) - :point (org-roam-reflink-point reflink) - :properties (org-roam-reflink-properties reflink))) - (insert ?\n))))) + (when-let ((refs (org-roam-node-refs node)) + (reflinks (seq-sort #'org-roam-reflinks-sort (org-roam-reflinks-get node)))) + (magit-insert-section (org-roam-reflinks) + (magit-insert-heading "Reflinks:") + (dolist (reflink reflinks) + (org-roam-node-insert-section + :source-node (org-roam-reflink-source-node reflink) + :point (org-roam-reflink-point reflink) + :properties (org-roam-reflink-properties reflink))) + (insert ?\n)))) ;;;; Grep (defvar org-roam-grep-map