NYC Data Hackers
Democratizing Data Science Teams With Quarto

boB Rudis (@hrbrmstr)

GreyNoise Intelligence

2022-10-26

Where To Find Me

rud.is/b

The Daily Drop



Data-Driven Security

@hrbrmstr
@hrbrmstr
𐍈 @hrbrmstr

• Enable team members to research/experiment/work the way they’re most comfortable/productive.
• Unify & level-up artifacts intended for humans.
• Finally achieve reproducibility goals.
• Use what the 😎 kids are using.

“Data Science” At GreyNoise

• Planetary Scale Passive Sensor Data
  • group_by(…) |> summarise(…)
  • Fundamental time-series analytics
  • “Fancy” ML/AI
• Website/Blog Performance
• Business Intelligence
• Sales Enablement/Insights
• Any question my boss/🏛/our users+community asks

Data “Science” Environments Used At GreyNoise

• Golang programs
• Jupyter notebooks
• Python scripts
• R Markdown documents
• R scripts
• [Experimenting with] Rust
• Scala (Glue)
• SQL

• Jupyter[Hu|La]b
• Observable / JavaScript
• RAWGraphs
• RStudio
• Sublime Text
• *vi*
• Visual Studio Code

A.K.A. Chaos

flowchart LR
    A[Pandas/Python] --> B[Jupyter]
    A[Pandas/Python] --> C[R Markdown]
    A[Pandas/Python] --> D[Shiny]
    G[R/Tidyverse] --> B[Jupyter]
    G[R/Tidyverse] --> C[R Markdown]
    G[R/Tidyverse] --> D[Shiny]
    L[Rust/Polars] --> B[Jupyter]
    L[Rust/Polars] --> E[Folder/Git]
    F[SQL] --> B[Jupyter]
    F[SQL] --> C[R Markdown]
    B[Jupyter] --> E[Folders/Git]
    C[R Markdown] --> E[Folders/Git]
    D[Shiny] --> I[Web]
    B[Jupyter] --> J[Lab/Hub]
    J[Lab/Hub] --> I[Web]
    C[R Markdown] --> I[Web]
    E[Folders/Git] --> I[Web]
    J[Lab/Hub] --> K[Pasted into gDocs/Word/Confluence...]
    C[R Markdown] --> K[Pasted into gDocs/Word/Confluence...]
    E[Folders/Git] --> K[Pasted into gDocs/Word/Confluence...]

1452–53

Knowledge Democratization

Quarto’s Organizational Sweet Spot: Documentation

Quarto’s Organizational Sweet Spot: Communication

Quarto’s Organizational Sweet Spot: Uniform Style

Quarto’s Organizational Sweet Spot: Reproducibility

🔋🔋🔋 Included

1st Class Citizens:

• R
• Python
• Julia
• Observbable/JavaScript #WAT
• But, I’m lying to you… R & Python have built-in extensibility (i.e. {V8}, {rJava}, {reticulate})
• And, it goes way beyond that…

So. Many. Kernels.¹

wordCloud = {
  const fontFamily = "Roboto Condensed, Arial, Helvetica, sans-serif";
  
  const svg = d3.create("svg")
    .attr("id", "word-cloud")
    .attr("viewBox", [0, 0, width, height])
    .attr("font-family", fontFamily)
    .attr("text-anchor", "middle");

  const displaySelection = svg.append("text")
    .attr("font-family", "Lucida Console, Courier, monospace")
    .attr("text-anchor", "start")
    .attr("alignment-baseline", "hanging")
    .attr("x", 10)
    .attr("y", 10);
    
  const cloud = d3.cloud()
    .size([width, height])
    .words(data.map(d => Object.create(d)))
    .padding(0)
    .rotate(() => 0)
    .font(fontFamily)
    .fontSize(d => s(d.value))
    .on("word", ({size, x, y, rotate, text}) => {
      svg.append("text")
        .attr("font-size", size)
        .attr("transform", `translate(${x},${y}) rotate(${rotate})`)
        .text(text)
        .classed("click-only-text", true)
        .classed("word-default", true)
        .on("mouseover", handleMouseOver)
        .on("mouseout", handleMouseOut)
        .on("click", handleClick);
      
        function handleMouseOver(d, i) {
          d3.select(this)
            .classed("word-hovered", true)
            .transition(`mouseover-${text}`).duration(200).ease(d3.easeLinear)
              .attr("font-size", size + 2)
              .attr("font-weight", "bold");
        }
        
        function handleMouseOut(d, i) {
          d3.select(this)
            .classed("word-hovered", false)
            .interrupt(`mouseover-${text}`)
              .attr("font-size", size);
        }
        
        function handleClick(d, i) {
          var e = d3.select(this);
          displaySelection.text(`selection="${e.text()}"`);
          e.classed("word-selected", !e.classed("word-selected"));
        }

    });
  
  cloud.start();
  invalidation.then(() => cloud.stop());
  
  return svg.node();
}

html`<style type="text/css">

  .click-only-text {
      cursor: pointer;
      -webkit-user-select: none;
      -moz-user-select: none;
      -ms-user-select: none;
      user-select: none;
    }
  .click-only-text::selection {
      background: none;
    }

  .word-default {
      fill: cadetblue;
      font-weight: normal;
    }
  .word-hovered {
      fill: white;
      font-weight: bold;
    }
  .word-selected {
      fill: darkslategrey;
      font-weight: bold;
    }

</style>`;

s = d3
  .scaleSqrt()
  .domain([1, d3.max(data.map((d) => d.value))])
  .range([6, 102])

padding = 0

height = 700

data = [ { "text": "Micronaut", "value": 17 }, { "text": "Agda kernel", "value": 12 }, { "text": "Dyalog Jupyter Kernel", "value": 1 }, { "text": "Coarray-Fortran", "value": 2 }, { "text": "LFortran", "value": 18 }, { "text": "Ansible Jupyter Kernel", "value": 3 }, { "text": "sparkmagic", "value": 12 }, { "text": "sas_kernel", "value": 9 }, { "text": "IPyKernel", "value": 17 }, { "text": "IJulia", "value": 18 }, { "text": "IHaskell", "value": 1 }, { "text": "IRuby", "value": 5 }, { "text": "tslab", "value": 15 }, { "text": "IJavascript", "value": 80 }, { "text": "ITypeScript", "value": 8 }, { "text": "jpCoffeescript", "value": 5 }, { "text": "jp-LiveScript", "value": 8 }, { "text": "Juka", "value": 14 }, { "text": "ICSharp", "value": 3 }, { "text": "IRKernel", "value": 54 }, { "text": "SageMath", "value": 9 }, { "text": "pari_jupyter", "value": 19 }, { "text": "IFSharp", "value": 15 }, { "text": "lgo", "value": 17 }, { "text": "iGalileo", "value": 20 }, { "text": "gopherlab", "value": 16 }, { "text": "Gophernotes", "value": 49 }, { "text": "IGo", "value": 73 }, { "text": "IScala", "value": 1 }, { "text": "almond (old name: Jupyter-scala)", "value": 5 }, { "text": "IErlang", "value": 4 }, { "text": "ITorch", "value": 14 }, { "text": "IElixir", "value": 13 }, { "text": "ierl", "value": 4 }, { "text": "IAldor", "value": 11 }, { "text": "IOCaml", "value": 5 }, { "text": "OCaml-Jupyter", "value": 12 }, { "text": "IForth", "value": 1 }, { "text": "peforth", "value": 2 }, { "text": "IPerl", "value": 5 }, { "text": "Perl6", "value": 12 }, { "text": "IPerl6", "value": 12 }, { "text": "Jupyter-Perl6", "value": 6 }, { "text": "IPHP", "value": 10 }, { "text": "Jupyter-PHP", "value": 17 }, { "text": "IOctave", "value": 4 }, { "text": "IScilab", "value": 6 }, { "text": "MATLAB Kernel", "value": 7 }, { "text": "Bash", "value": 17 }, { "text": "Z shell", "value": 8 }, { "text": "Pharo Smalltalk", "value": 9 }, { "text": "PowerShell", "value": 9 }, { "text": "CloJupyter", "value": 2 }, { "text": "CLJ-Jupyter", "value": 15 }, { "text": "jupyter-kernel-jsr223", "value": 9 }, { "text": "Hy Kernel", "value": 2 }, { "text": "Calysto Hy", "value": 11 }, { "text": "Redis Kernel", "value": 19 }, { "text": "jove", "value": 7 }, { "text": "jp-babel", "value": 20 }, { "text": "ICalico", "value": 16 }, { "text": "IMathics", "value": 13 }, { "text": "IWolfram", "value": 17 }, { "text": "Lua Kernel", "value": 1 }, { "text": "IPurescript", "value": 15 }, { "text": "IPyLua", "value": 19 }, { "text": "ILua", "value": 7 }, { "text": "Calysto Scheme", "value": 14 }, { "text": "Calysto Processing", "value": 7 }, { "text": "idl_kernel", "value": 10 }, { "text": "Mochi Kernel", "value": 6 }, { "text": "Lua (used in Splash)", "value": 5 }, { "text": "Apache Toree (formerly Spark Kernel)", "value": 2 }, { "text": "Skulpt Python Kernel", "value": 42 }, { "text": "Calysto Bash", "value": 11 }, { "text": "MetaKernel Python", "value": 71 }, { "text": "IVisual", "value": 12 }, { "text": "IBrainfuck", "value": 15 }, { "text": "JupyterQ (KX Official Kernel)", "value": 9 }, { "text": "KDB+/Q Kernel (IKdbQ)", "value": 20 }, { "text": "KDB+/Q Kernel (KdbQ Kernel)", "value": 6 }, { "text": "PyQ Kernel", "value": 12 }, { "text": "ICryptol", "value": 2 }, { "text": "cling", "value": 6 }, { "text": "xeus-cling", "value": 8 }, { "text": "Xonsh", "value": 16 }, { "text": "Prolog", "value": 6 }, { "text": "SWI-Prolog", "value": 19 }, { "text": "common-lisp-jupyter", "value": 15 }, { "text": "Maxima-Jupyter", "value": 15 }, { "text": "ielisp", "value": 2 }, { "text": "Calysto LC3", "value": 13 }, { "text": "Yacas", "value": 15 }, { "text": "IJython", "value": 9 }, { "text": "ROOT", "value": 16 }, { "text": "Gnuplot Kernel", "value": 10 }, { "text": "Tcl", "value": 12 }, { "text": "Tcl", "value": 6 }, { "text": "J", "value": 16 }, { "text": "Jython", "value": 5 }, { "text": "C", "value": 13 }, { "text": "jupyterC", "value": 8 }, { "text": "TaQL", "value": 5 }, { "text": "Coconut", "value": 12 }, { "text": "SPARQL", "value": 2 }, { "text": "AIML chatbot", "value": 13 }, { "text": "IArm", "value": 11 }, { "text": "SoS", "value": 4 }, { "text": "jupyter-nodejs", "value": 4 }, { "text": "Pike", "value": 11 }, { "text": "imatlab", "value": 6 }, { "text": "jupyter-kotlin", "value": 12 }, { "text": "jupyter_kernel_singular", "value": 9 }, { "text": "spylon-kernel", "value": 20 }, { "text": "mit-scheme-kernel", "value": 2 }, { "text": "elm-kernel", "value": 20 }, { "text": "Isbt", "value": 1 }, { "text": "BeakerX", "value": 13 }, { "text": "MicroPython", "value": 80 }, { "text": "IJava", "value": 55 }, { "text": "Guile", "value": 17 }, { "text": "circuitpython_kernel", "value": 18 }, { "text": "stata_kernel", "value": 3 }, { "text": "iPyStata", "value": 16 }, { "text": "IRacket", "value": 55 }, { "text": "jupyter-dot-kernel", "value": 17 }, { "text": "Teradata SQL kernel and extensions", "value": 15 }, { "text": "HiveQL Kernel", "value": 10 }, { "text": "EvCxR Jupyter Kernel", "value": 5 }, { "text": "StuPyd Kernel", "value": 5 }, { "text": "coq_jupyter", "value": 15 }, { "text": "Cadabra2", "value": 14 }, { "text": "iMongo", "value": 14 }, { "text": "jupyter_kernel_chapel", "value": 11 }, { "text": "A Jupyter kernel for Vim script", "value": 13 }, { "text": "SSH Kernel", "value": 14 }, { "text": "GAP Kernel", "value": 1 }, { "text": "Wolfram Language for Jupyter", "value": 13 }, { "text": "GrADS kernel", "value": 1 }, { "text": "Bacatá", "value": 4 }, { "text": "nelu-kernelu", "value": 9 }, { "text": "IPolyglot", "value": 17 }, { "text": "Emu86 Kernel", "value": 8 }, { "text": "Common Workflow Language (CWL) Kernel", "value": 7 }, { "text": "MIPS Jupyter Kernel", "value": 13 }, { "text": "iTTS", "value": 3 }, { "text": "xeus-clickhouse", "value": 14 }, { "text": "IQSharp", "value": 1 }, { "text": ".Net Interactive", "value": 18 }, { "text": "mariadb_kernel", "value": 12 }, { "text": "ISetlX", "value": 15 }, { "text": "Ganymede", "value": 13 }, { "text": "cqljupyter", "value": 2 }, { "text": "iclingo", "value": 2 }, { "text": "ICrystal", "value": 17 }, { "text": "crystal_kernel", "value": 1 }, { "text": "idg", "value": 16 }, { "text": "Whitenote", "value": 7 } ]

d3 = {
  const d3 = await require("d3@5", "d3-array@2");
  d3.cloud = await require("d3-cloud@1");
  return d3;
}

1. https://github.com/jupyter/jupyter/wiki/Jupyter-kernels

Observable, You Say?¹

1. https://observablehq.com/

Observable, You Say?¹

1. https://chrome.google.com/webstore/detail/quartize/bcggojdjahdjhccmeanbnidmoicknmbb

Reproduciblity

quarto render …

The Quarto Value Proposition (i.e. What you get…)

• Access to language agnostic cutting-edge data analysis and visalization tools
• Uniform styling across web sites, documents, and presenations
• Reproduciblity
• Content Adaptibility
• Fame and fortune

Where To Start

• 🤓 quarto.org
• ✨ Awesome Quarto 🔖
• 👩🏽‍🎨 Your org/team designer/brand folks
• 💪 Creating a Website
  • 🥶 Freezing content
• 🏋🏽‍♂️ Rocker
  • ➕ Extending Docker

Some Of My Quarto Resources

• rstudio-electron-quarto-installer
• quarto-organization-template
• quarto-r-ojs-choropleth
• quarto-538-house-carto
• conus-foliage-2022
• ohq2quarto

A.M.A.

Quarto RevealJS Source:
https://github.com/hrbrmstr/2022-10-nycdh