47 Degrees joins forces with Xebia read more

How org works

How org works

I recently wrote an introduction to org and how you can use it to generate a website for your GitHub organization. Now, it’s time to dive into the technical details of how org works.

org is written in Clojure and ClojureScript although most of its code is compatible with both languages. This means that we can run the UI components or the data fetching code in the JVM or JS environments. It has some platform-specific code for file IO in the JVM and mounting the application in the DOM, but most of the code can run in the client and the server. The project also utilizes some amazing technologies such as React, and a few interesting Clojure libraries.

UI

The UI is rendered with React although org uses an excellent React wrapper library called Rum, which lets you write components in a Clojure DSL that translates to React components. In the following example, we use Rum’s defc macro for defining a component for displaying some stats about our organization, notice that it looks like a function:

(rum/defc stats
  [repos]
  [:div.github-stats
   [:ul
    [:li.contributors
     [:span (sum-by :contributors repos)]
     [:span [:span.octicon.octicon-person] "contributors"]]
    [:li.stars
     [:span (sum-by :stars repos)]
     [:span [:span.octicon.octicon-star] "stars"]]
    [:li.repositories
     [:span (count repos)]
     [:span [:span.octicon.octicon-repo] "repositories"]]
    [:li.languages
     [:span (count (all-languages repos))]
     [:span [:span.octicon.octicon-code] "languages"]]]])

This is a simple component that takes a collection of repos and shows statistical information about them. Another example would be the search component, which receives the app state atom as a parameter and stores the query the user has entered in the search field.

(rum/defc search
  [state]
  [:div.search
   [:input
    {:type "text"
     :name "nombre"
     :placeholder "Search a project"
     :on-change (fn [ev]
                  (swap! state assoc :query (.-value (.-target ev))))}]])

A nice property of Rum components is that they can be rendered both in JS environments (by React) or in the JVM, so writing the UI with Rum gives us prerendering for free.

(require '[rum.core :as rum])

(rum/defc hello
  [name]
  [:p (str "Hello, " name)])

In the browser, we would mount the application in a DOM node:

(rum/mount (hello "World") (js/document.getElementById "app"))

and in the JVM, we can just render it to a static HTML. If the HTML we are generating is going to be static, we use render-static-markup and get an HTML string. If the HTML we are generating is going to be handled by React in the client, we use render-html to include react metadata and avoid expensive checks in the client:

(rum/render-static-markup (hello "World"))
;; => "<p>Hello, World</p>"

(rum/render-html (hello "World"))
;; => "<p data-reactroot=\"\" data-reactid=\"1\" data-react-checksum=\"-122613252\">Hello, World</p>"

Data fetching

Since org is a tool for generating a site for a GitHub organization, it has to interact with the GitHub API for fetching data. There is an experimental GraphQL endpoint for GitHub but it’s still experimental and unable to serve the data we need in org, so we’re stuck using the traditional REST API. There are a few nuances in fetching data such as pagination, and a proliferation of endpoints for fetching every project’s information.

HTTP client

The HTTP library of choice is httpurr, which lets us share the HTTP client code between Clojure and ClojureScript and has a promise-based API. The following is an example of how we fetch an organization’s repositories:

(require '[httpurr.client :as http])
(require '[httpurr.status :as status])
(require '[promesa.core :as p])

;; in Clojure:       (require '[httpurr.client.aleph :refer [client]])
;; in ClojureScript: (require '[httpurr.client.xhr :refer [client]]))

(defn parse-response
  [{:keys [body]}]
  ;; ...
  )
   
(defn get-org-repos!
  [org token]
  (let [request {:method :get
                 :url (str "https://api.github.com/orgs/" org "/repos")
                 :query-string "type=public&per_page=100"
                 :headers {"authorization" "Token a-github-token"}}]
    (p/then (http/send! client request)
            (fn [resp]
              (if (status/success? resp)
	            (parse-response resp)
                (p/rejected (ex-info "Unsuccessful request" {:response resp})))))))

The above get-org-repos! function performs a GET request to GitHub to obtain an organization’s repositories. In httpurr requests are represented as plain maps, and we perform them with a platform-specific client, backed by aleph in Clojure and Xhr in ClojureScript. The library returns a platform-agnostic promise/future type which uses Java8’s CompletableFuture in Clojure and bluebird promises in ClojureScript, backed by the promesa library.

A request to fetch a project’s language information looks like the following:

(defn parse-langs-response
  [{:keys [body]}]
  ;; ...
  )
  
(defn get-languages!
  [user repo token]
  (let [request {:method :get
                 :url (str "https://api.github.com/repos/" user "/" repo "/languages")
                 :headers {"authorization" "Token a-github-token"}}]
    (p/then (http/send! client request)
            (fn [resp]
              (if (status/success? resp)
	            (parse-langs-response resp)
                (p/rejected (ex-info "Unsuccessful request" {:response resp})))))))

It’s not sufficient to just fetch all of the projects for an organization, we also need to get the repo languages and each project’s contributors. While we wait for GraphQL support in the GitHub API we’ll need to perform a few requests per project to get all the information we want to show.

It seems we can optimize data access by performing some requests in parallel, so we’ll use urania, a library inspired by Haxl that works both in Clojure and ClojureScript.

Optimizing data access

We’ll need to tell urania how to fetch every piece of data: organization repos, repo languages, and repo contributors. To do so, we simply create a data type and make it implement the DataSource protocol.

(require '[urania.core :as u])

(deftype OrgRepos [org]
  u/DataSource
  (-identity [_]
    [:repos org])
  (-fetch [_ {:keys [token]}]
    (get-org-repos! org token)))
	
(deftype Languages [repo]
  u/DataSource
  (-identity [_]
    [:languages repo])
  (-fetch [_ {:keys [token]}]
    (get-languages! repo)))
	
(deftype Contributors [repo]
  u/DataSource
  (-identity [_]
    [:contributors repo])
  (-fetch [_ {:keys [token]}]
    (get-contributors! repo)))

We can then use urania’s combinators on values that implement DataSource to build a “recipe” of the data fetching.

(defn fetch-langs-and-contribs
  [repo]
  (u/map
   (fn [[languages contributors]]
     (assoc repo :languages languages :contributors contributors))
   (u/collect [(Languages. repo)
               (Contributors. repo)])))

(def fetch-data
  (u/traverse
    fetch-langs-and-contribs
	(OrgRepos. "47deg")))

And execute it passing configuration such as the GitHub token, we’ll get back a promise resolved with the final value:

(u/run! fetch-data {:env {:token "your-github-token"}})

When running it, this will:

  • Make one request to get all the organization repos
  • For each repo, concurrently:
    • get its languages
    • get its contributors

minimizing the latency cost of making all of those HTTP requests. A key difference between running the data-fetching code in the JVM or in JS environments is that in the JVM we can block the current thread to wait for the requests to finish. We take advantage of that for prerendering the site.

Configuration

Since org builds a website from a configuration file, a way to validate and document configuration is needed. For this purpose, I used the Clojure spec library, which allows us to define specifications for the configuration map that appear as follows:

(require '[clojure.spec :as s])

(s/def :org/config (s/keys :req-un [:org/organization
                                    :org/logo
                                    :org/links
                                    :org/languages
                                    :org/included-projects
                                    :org/style]
                           :opt-un [:org/organization-name
                                    :org/social
                                    :org/extra-repos
                                    :org/project-logos]))

The above specs describe the configuration as a map with some required keys (:organization, :logo, :links and such) and other optional ones (:organization-name, :social, :extra-repos, and :project-logos). We then spec each individual attribute separately, for instance, the organization name and logo look like the following:

(s/def :org/organization string?)

(s/def :logo/href string?)
(s/def :logo/src string?)
(s/def :org/logo (s/keys :req-un [:logo/href
                                  :logo/src]))

The organization name must be a string, and the logo a map with :href and :src string attributes. Let’s look how spec helps us validate configuration and display error messages:

(s/valid? :org/logo {:href "https://47deg.com" :src "img/nav-brand.png"})
;; => true

(s/valid? :org/logo {:href "https://47deg.com"})
;; => false

(s/explain :org/logo {:href "https://47deg.com"})
val: {:href "https://47deg.com"} fails spec: :org/logo predicate: (contains? % :src)
;; => nil

We use spec’s valid? predicate with a spec and a value to check whether the value conforms to the specification. The explain function checks if the value conforms to the spec and reports any failures it finds. Spec can do much more, but these features make it perfect for checking the validity of the configuration and providing helpful error messages.

Conclusion

By leveraging great open source technologies we were able to quickly build a website generator with server-side rendering, customization through configuration with friendly error messages, and optimized data fetching. Shout out to the authors of the libraries I’ve pulled together for building org; it wouldn’t be possible without standing on the shoulders of giants.

Ensure the success of your project

47 Degrees can work with you to help manage the risks of technology evolution, develop a team of top-tier engaged developers, improve productivity, lower maintenance cost, increase hardware utilization, and improve product quality; all while using the best technologies.