chrisdinger.com

Controlling my garage door with Elixir and Nerves

Thu, 25 Feb 2021 00:00:00 +0000

I’ve wanted to learn Elixir for years now and have been looking for a little project to explore with. The best way to learn a new language for me is to solve a concrete problem I have. Toy apps don’t work for me.

I’ve also been following the Nerves project and am really interested in using Elixir/Erlang OTP model for embedded projects. Elixr/Erlang’s expect failures and deal with them mantra really fits well with the embedded world. I’ve dabbled with embedded development for a long time and have always preferred to reach for C. It’s super lightweight and tiny, but also super easy to screw up. With the availability and low cost of more powerful systems like Raspberry Pi Zero, you can more easily afford to move up to a higher-level language like Elixir without making too many trade-offs (yes I know MicroPython exists).

Automating my garage door seems like the perfect excuse to finally do something with Elixir and Nerves. I can’t see my garage door from inside my house and I’m always wondering if it’s open or closed. I could buy a $50 retail sensor/switch to do this, but that’s not fun and it won’t help me learn Elixir. I’ve also used a homebridge setup to do this in the past, but the homebridge ecosystem is maddening. Any given plugin has at least a dozen variants and finding the one that’s been updated in the past decade is a chore.

Hardware

Raspberry Pi Zero W (the one with wireless networking)
5-volt relay module
Magnetic reed switch

Software

Elixir and Nerves for firmware
Home Assistant
MQTT server (via Home Assistant)

Firmware

Let’s start with the sensor. Something that will tell me if the garage door is open or closed. The simple way to do this is with a magnetic reed switch. It’s a switch that has two contacts and a long thin metal reed and opens and closes the switch in the presence of a strong magnet. I mounted the sensor on the frame of the garage door and the magnet onto the door itself.

The one I’m using has screw terminals for NC (normally closed) and NO (normally open) configurations. I’m using it in NO mode so the system will fail open. That way if the wires become disconnected for any reason, it will indicate that the door is open. What I don’t want is for something to fail and indicate the the door is closed when it isn’t.

Nerves has CircuitsGPIO for interacting with your board’s general purpose input/output pins; that’s what we’ll use here. Install it by adding it to your mix dependencies:

  {:circuits_gpio, "~> 0.4"}

I’ve created an Elixir module called Garage.DoorSensor that implements the GenServer callbacks required for this. The GenServer abstraction is always the piece I’ve struggled with in Elixir. Here’s the minimum setup needed to sense the door state. It’s all GenServer callbacks:

defmodule Garage.DoorSensor do
  use GenServer
  require Logger

  @sensor_pin 26

  def start_link(_) do
    GenServer.start_link(__MODULE__, [], name: __MODULE__)
  end

  def init(_) do
    {:ok, gpio} = Circuits.GPIO.open(@sensor_pin, :input, pull_mode: :pullup)
    Circuits.GPIO.set_interrupts(gpio, :both)
    {:ok, %{gpio: gpio}}
  end

  def handle_info({:circuits_gpio, @sensor_pin, _timestamp, value}, state) do
    Logger.info("Garage door is #{value}")
    {:noreply, state}
  end
end

The init funciton is called when the GenServer process starts. It sets up the pin our sensor is connected to and configures an interrupt (a function that will be called when the pin state changes) to handle the state changes. What isn’t super clear here for Elixir/Nerves newcomers is that the interrupt is a GenServer info message. This isn’t explicitly documented anywhere. I had to look at the CircuitsGPIO examples and search though other Github repositories to sort this out.

I’ve connected to two leads of the sensor to pin 26 and ground. When the switch/door is open, pin 26 is connected to ground.

Activating the opener

Controlling the door opener relay is a lot simpler—it’s just a function call (Circuits.GPIO.write). I did wrap this up into a GenServer module in order to store the GPIO reference:

defmodule Garage.DoorSwitch do
  use GenServer
  require Logger

  @switch_activation_time_in_ms 500
  @switch_pin 4

  def activate do
    GenServer.cast(__MODULE__, :activate)
  end

  def start_link(opts) do
    GenServer.start_link(__MODULE__, opts, name: __MODULE__)
  end

  def init(_state) do
    Logger.info("Door switch init...")
    {:ok, gpio} = Circuits.GPIO.open(@switch_pin, :output)
    {:ok, %{gpio: gpio}}
  end

  def handle_cast(:activate, state = %{gpio: gpio}) do
    Logger.info("Activating door switch...")
    Circuits.GPIO.write(gpio, 1)
    Process.sleep(@switch_activation_time_in_ms)
    Circuits.GPIO.write(gpio, 0)

    {:noreply, state}
  end
end

There’s also the on/off timing logic. We can’t just turn the relay on—that’d be like mashing down the opener button and never letting go. We want to simulate a button press. I chose 500ms (half a second) and it seems to work fine.

There are more GenServer callbacks here. Just like in Garage.DoorSensor, we have start_link and init. In this module we’re using GenServer.cast because we don’t need a response. It’s just call-and-forget.

The other difference here is that we’ve added a public API—the activate function. This is the GenServer pattern you’ll encounter: define public API functions and implement them in GenServer callbacks using the appropriate callback flavor (call, cast, and info).

Tying it all together

The other bit of GenServer that has confused me in the past is how to start these processes up. My poor old programmer brain sees the world in OO. I’ve been thinking of these Elixir processes as object instances. You “instantiate” them and send them messages, right? So where do these things get intstantiated? The supervision tree. Your generated Nerves project skeleton will come an application file (Garage.Application in my case). This is where these child processes are defined and configured. Here’s an abbreviated version of what I’m using:

defmodule Garage.Application do
  use Application

  def start(_type, _args) do
    opts = [strategy: :one_for_one, name: Garage.Supervisor]

    children =
      [
        {Garage.DoorSensor, []},
        {Garage.DoorSwitch, []}
      ] ++ children(target())

    Supervisor.start_link(children, opts)
  end
end

RADIUS authentication in SQL Developer on MacOS (updated)

Mon, 02 Jul 2018 00:00:00 +0000

This post is an update to this original overview, which details configuring SQL Developer to perform RADIUS authentication on MacOS. This updated post contains newer versions numbers and simpler instructions.

1. Install instantclient

Download instantclient 12.2.0.1.0 from Oracle. You’ll only need the basic package. Unzip this package and place it in an accessible location. I install my instant client in ~/lib/instantclient_12_2. You can install yours wherever you like.

Homebrew users: the Homebrew packages available strip out the .jar files which confuses SQL Developer; download this library directly from Oracle.

2. sqlnet.ora

The OCI driver properties can be configured via a special file called sqlnet.ora. The property we’re interested in is SQLNET.AUTHENTICATION_SERVICES.

Create a sqlnet.ora file and place it in a directory you’ll reference later as TNS_ADMIN. I put mine alongside the instantclient directory in step 1: ~/lib/sqlnet.ora.

Make sure the sqlnet.ora file has an entry for radius:

SQLNET.AUTHENTICATION_SERVICES=(none,radius)

3. Mangle the java.library.path property

Modern MacOSes—version 10.11 (El Capitan) and newer—have a feature called System Integrity Protection (SIP) which ignores the DYLD_LIBRARY_PATH environment variable. This is how the instantclient path is typically revealed to SQL Developer. SQL Developer still looks at this environment variable even though we can no longer use it. It’s simply not an option for setting this path.

Instead we can clobber the java.library.path via a not-so-invasive configuration file, ~/.sqldeveloper/<VERSION>/product.conf. Open this file and set java.library.path to your instantclient path (the one you chose in step 1) at the very bottom. If you chose the same location I did, yours should look like this (replace <YOUR_USERNAME> with your username):

AddVMOption -Djava.library.path=/Users/<YOUR_USERNAME>/lib/instantclient_12_2

4. Set the TNS_ADMIN path for GUI applications

You must set the TNS_ADMIN environment variable to make this work. Note that this needs to be set in the GUI environment via launchctl—simply setting the environment variable in a terminal will not work. Unfortunately, values set via launchctl don’t persist between sessions (if you logout or restart, the value is lost).

In order to set this value upon start, we’ll create the file ~/.sqldeveloper/env.sh. SQL Developer uses this file to set environment variables upon startup.

Create/open this file and add a reference to your TNS_ADMIN location:

launchctl setenv TNS_ADMIN /Users/<YOUR_USERNAME>/lib

Note that TNS_ADMIN should point to a directory—not a file.

4. Configure SQL Developer to use OCI

Now, fire up SQL Developer. Navigate to Oracle SQL Developer > Preferences and choose Database > Advanced:

Check the Use Oracle Client and Use OCI/Thick driver checkboxes.

Then, click Configure and enter the location of your instantclient:

Make sure you change the Client type to Instant Client.

You may click the Test button here if you want, though I’ve found that its output is not always useful (failures here do not necessarily indicate a broken configuration).

5. Verify

In SQL Developer, go to Oracle SQL Developer > About Oracle SQL Developer. Click on the Properties tab and search for the sqldeveloper.oci.available property. If the value is true, everything is set up correctly. You should now be able to authenticate using RADIUS!

References

How I learned to stop whining and tolerate my touchbar MacBook Pro

Thu, 06 Jul 2017 00:00:00 +0000

I was nervous about ordering my new touchbar-enhanced MacBook Pro. As a long-time vim user I was already sold on the value of keeping my eyes on the screen and off the keyboard. I couldn’t be less interested in glancing down to tap a distorted rendering of a Safari tab or an OK button on a dialog. But mostly I was worried about losing my precious escape key.

Once I got the machine and started using it, I found it to be even more awkward than I anticipated. After a several weeks of desperately groping for a phantom escape key I’d had enough. I knew of two physical escape key alternatives: remapping cap lock and ye olde crtl+[. The most realistic for me was to re-purpose the caps lock key (which I often completely disable anyway). In MacOS, this is expectedly simple: navigate to System Preferences > Keyboard > Modifier Keys. Just change the Caps Lock Key option to Escape and you’re good to go.

I had wanted to try this in the past but was very concerned about re-training my fingers. After going through with this, I’m happy to report that it only took a day—maybe two—to become comfortable with the new configuration.

RADIUS authentication in SQL Developer on MacOS

Fri, 24 Feb 2017 00:00:00 +0000

There’s an updated, simpler version of this post that you’ll want to use for current versions of SQL Developer.

This post will walk you through configuring SQL Developer to perform RADIUS authentication on MacOS. I periodically need to do this to connect to databases that require multi-factor authentication (MFA) and I struggle with it every time. More than anything, this is just yet another entry in my Oracle Survival Guide to help future-me waste less time. Hopefully it helps others too!

A note on versions

Version matters here. While SQL Developer will let you point to any instant client you like, it’ll secretly be expecting a specific version that depends on which version of SQL Developer you’re using. If you don’t point it to that exact version of Instant Client, things won’t work—and it won’t be clear why.

As of the writing of this post, the latest versions are SQL Developer 4.2.0 and Instant Client 12.1.0.2. That’s what I’ll use in this guide.

Download these resources directly from Oracle. The Homebrew packages available strip out the .jar files which confuses SQL Developer.

1. Install instantclient

Download instantclient 12.1.0.2 from Oracle. You’ll only need the basic package. Unzip this package and place it in an accessible location. I install my instant client in ~/lib/instantclient_12_1. You can install yours wherever you like.

Oracle delivers this library to you in a broken state. Your instantclient contains a dynamic library called libclntsh.dylib.12.1, but every bit of software you’ll want to use will be looking for libclntsh.dylib. cd into your instantclient directory and ln -s libclntsh.dylib.12.1 libclntsh.dylib to un-broken it.

2. sqlnet.ora

The OCI driver properties can be configured via a special file called sqlnet.ora. The property we’re interested in is SQLNET.AUTHENTICATION_SERVICES.

Create a sqlnet.ora file and place it in a directory you’ll reference later as TNS_ADMIN. Make sure sqlnet.ora file has an entry for radius:

SQLNET.AUTHENTICATION_SERVICES=(none,radius)

3. Mangle the java.library.path property

Instead we can clobber the java.library.path via a not-so-invasive configuration file, ~/.sqldeveloper/4.2.0/sqldeveloper.conf. Create this file and set java.library.path to your instantclient path:

AddVMOption -Djava.library.path=/Users/ding0057/lib/instantclient_12_1

4. Set the TNS_ADMIN path for GUI applications

In order to set this value upon start, create a ~/Library/LaunchAgents/oracle.plist file that looks like this (replace the path with your own):

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>Label</key>
    <string>Set TNS_ADMIN at login</string>
    <key>ProgramArguments</key>
    <array>
      <string>sh</string>
      <string>-c</string>
      <string>launchctl setenv TNS_ADMIN /directory/where/sqlnet.ora/lives</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
  </dict>
</plist>

Note that TNS_ADMIN should point to a directory—not a file.

You’ll need to logout or restart for this change to become active.

4. Configure SQL Developer to use OCI

Now, fire up SQL Developer. Navigate to Oracle SQL Developer > Preferences and choose Database > Advanced:

Check the Use Oracle Client and Use OCI/Thick driver checkboxes.

Then, click Configure and enter the location of your instantclient:

Don’t worry about clicking the Test button here. Its output is not useful (failures here do not necessarily indicate a broken configuration). Even when everything is working this test fails—ignore it.

5. Verify

References

Postgres foreign data wrapper for Oracle (oracle_fdw) on MacOS

Tue, 20 Dec 2016 00:00:00 +0000

The Ecto adapter landscape currently lacks an Oracle option. If you’re itching to try out Elixir and/or Phoenix but really need to connect to an Oracle database to do anything interesting, you may want to give Laurenz Albe’s oracle_fdw project a try. oracle_fdw implements a foreign data wrapper for Postgres that allows you to connect to, read from, and manipulate remote Oracle databases.

Oracle_fdw relies on DYLD_LIBRARY_PATH to properly link Oracle libraries at runtime, but MacOS version 10.11 (El Capitan) and newer ignore DYLD_LIBRARY_PATH as part of its System Integrity Protection (SIP) feature. These steps will help you both build and install oracle_fdw on MacOS.

1. Install Postgres

The easiest way to install Postgres is homebrew: brew install postgres. Once installed it can be started with brew services start postgres.

2. Install instantclient

Download an instantclient from Oracle. You’ll need both the basic package and the SDK package. Out of the box, oracle_fdw expects that you’ll be using version 11.2. If you want to use 12.1, you’ll need to create some symlinks to trick it.

I install my instant client in ~/lib/instantclient_11_2. You can install yours wherever you like—just be sure that you expose an environment variable called ORACLE_HOME that points to your installation.

3. Build oracle_fdw

Download the latest release of oracle_fdw and unpack it. cd into the unpacked directory and compile the extension with make.

Once oracle_fdw is compiled, you have two options for proceeding: modify the compiled binary or symlink your instantclient to a location it can find. Without doing one of these, you’ll likely encounter this error upon loading the extension in Postgres:

ERROR:  could not load library "/usr/local/lib/postgresql/oracle_fdw.so":
  dlopen(/usr/local/lib/postgresql/oracle_fdw.so, 10):
  Library not loaded: @rpath/libclntsh.dylib.11.1
  Referenced from: /usr/local/lib/postgresql/oracle_fdw.so
  Reason: image not found

Option 1: alter the oracle_fdw binary

This is the option I chose. It lets me tweak the compiled binary while leaving everything else as-is. After compiling with make (and before make install), issue this command:

install_name_tool -add_rpath $ORACLE_HOME oracle_fdw.so

This adds your ORACLE_HOME path to the list of rpaths in oracle_fdw.so.

Option 2: symlink libclntsh.dylib.11.1 to an expected location

The other option is to symlink libclntsh.dylib.11.1 to one of the default search paths, ~/lib:

ln -s $ORACLE_HOME/libclntsh.dylib.11.1 ~/lib/

4. Install oracle_fdw

Now that oracle_fdw is compiled and ready to link up the instantclient, install it and its friends with make install.

At this point you should be able to fire up your favorite Postgres client and enable the extension with this SQL:

CREATE EXTENSION oracle_fdw;

If you don’t see any gnarly errors, you’re in business! I’ll write up another post soon that demonstrates how to use the extension.

Using erloci to talk to Oracle in Elixir

Wed, 23 Nov 2016 00:00:00 +0000

One of the great things about Elixir is its clean Erlang interoperability. The adoption rate of new languages like Elixir depends greatly on the ecosystem of available libraries. While the Elixir ecosystem is growing, being able to lean on the twenty(ish) years of existing open source Erlang libraries helps a lot.

Here are some examples of interacting with an Oracle database from Elixir via the erloci Erlang library.

Connect

erloci requires you to specify connection details as a TNS string. This can be slightly annoying but you can easily interpolate host, port, and service name into a TNS-style format.

# connect
host = "somehost.yourorg.com"
port = "1521"
service = "someservice"
username = "username"
password = "password"
tns = "(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=tcp)(HOST=#{host})(PORT=#{port})))(CONNECT_DATA=(SERVICE_NAME=#{service})))"
oci = :erloci.new([])
session = oci.get_session(tns, username, password)

# DDL
create = session.prep_sql("create table blahs (id integer, name varchar(100))")
{:executed, 0} = create.exec_stmt

Select

All SQL interactions with erloci are proper prepared statements. The exec_stmt/0 function returns a list of column and type information. Data must be explicitly fetched with fetch_rows/1.

iex> sql = session.prep_sql("select * from blahs")
iex> {:cols, columns} = sql.exec_stmt
{:cols, [{"ID", :SQLT_NUM, 22, 38, 0}, {"NAME", :SQLT_CHR, 200, 0, 0}]}
iex> {{:rows, rows}, false} = sql.fetch_rows(1)
{{:rows,
  [[<<2, 193, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>,
    "ahoy!"]]}, false}

What the heck is <<2, 193, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0>>? Well, from the column information returned from exec_stmt we know it’s of type :SQLT_NUM but what erloci is returning is binary. You’ll want to convert it using :oci_util.oranumber_decode/1. In this case, it’s the integer 2. See the oci_util module for a pile of other handy conversion function and utilities.

Insert

erloci has full support for bound parameters. Parameters that are to be bound can be named (like :id or :name) or can use numbers (:1, :2, :3) in the SQL statement. bind_vars/1 must then be called to supply type information for each of the bound parameters. Finally, exec_stmt/1 is used to pass in values and execute the insert.

# insert with bound parameters
insert = session.prep_sql("insert into blahs (id, name) values (:id, :name)")
:ok = insert.bind_vars([{":id", :SQLT_INT}, {":name", :SQLT_CHR}])
{:rowids, ["AABUUnAQAAAEGgTAAA"]} = insert.exec_stmt([{1, "blahblah"}])

Errors

When errors are encountered erloci returns the Oracle error code and message.

create = session.prep_sql("create table blahs (id integer, name varchar(100))")
{:error, {955, "ORA-00955: name is already used by an existing object\n"}} = create.exec_stmt

The Roman Numerals kata

Thu, 07 Jul 2016 00:00:00 +0000

I recently read Sandi Metz’s Make Everything the Same post and her words really resonated with me. I too had recently completed the Roman Numerals kata in Elixir and, while I was happy with my solution, I was struck by how different it was from most others.

Special cases in disguise

Most solutions I explored had some flavor of this kind of configuration:

@numerals [
    [1000, "M"],
    [ 900, "CM"],
    [ 500, "D"],
    [ 400, "CD"],
    [ 100, "C"],
    [  90, "XC"],
    [  50, "L"],
    [  40, "XL"],
    [  10, "X"],
    [   9, "IX"],
    [   5, "V"],
    [   4, "IV"],
    [   1, "I"]
  ]

There are two things that bother me about this: the configuration contains duplication and it makes assumptions about the implementation that uses it.

The implementations that make use of this kind of configuration were indeed short (it really only take three more lines of code) and fast, but I found them to be opaque. When approaching the Roman Numerals kata, I tried to write a program in the way my brain processes the conversion instead of a clever mathematical solution. As Sandi says, code is read more than it’s written—so I prefer a solutions that are easily understandable.

To me, the only real needed configuration is this:

@numerals [
    [1000, "M"],
    [ 500, "D"],
    [ 100, "C"],
    [  50, "L"],
    [  10, "X"],
    [   5, "V"],
    [   1, "I"]
  ]

The four and nine cases are just special combinations of these base symbols. I’d argue that the first example contains special cases—they’re just in disguise.

Make everything the same, functionally

The next step for me was to abstract the application of these symbols for any given digit in a decimal number. A digit in a decimal number represents two things: a numeral and a magnitude (a 3 in the ones’ place is different than a 3 in the hundreds’ place). In Roman numeral notation the symbol changes at different magnitudes, but the pattern is always the same. This pattern is what I chose to abstract.

def template(digit, one, five, ten) do
  case digit do
  1 -> one
  2 -> one  <> one
  3 -> one  <> one <> one
  4 -> one  <> five
  5 -> five
  6 -> five <> one
  7 -> five <> one <> one
  8 -> five <> one <> one <> one
  9 -> one  <> ten
  _ -> ""
end

Give the template a digit and the Roman symbols for that magnitude, and it gives you the Roman numeral representation of that digit.

The magnitude here is simply the position of the Arabic numeral you’re currently processing:

def magnitude(decimal_place) do
  :math.pow(10, decimal_place) |> round
end

And then that magnitude maps nicely onto our configuration to reveal the Roman symbols the template is after.

template(digit, @numerals[1 * magnitude],
                @numerals[5 * magnitude],
                @numerals[10 * magnitude])

Now yes, the template function contains the verboten case statement, but its intention here is not to branch off to other behaviors in the code. It’s simply formatting the supplied numerals. To me at least, this feels different.

My most recent version is this solution and I’m mostly happy with it. Though I’m already tempted to revisit this kata. It’d be interesting to refactor away the case statement altogether, which might teach me a new thing or two about Elixir in the process. I suppose that’s the point!

Self-sourcing bash profile

Thu, 23 Jul 2015 00:00:00 +0000

If you’re like me, you’re constantly popping into ~/.bash_profile (or it’s friends) and making tweaks. I’ve always kept an alias for this, eb. That makes the edit quicker, but the obnoxious source ~/.bash_profile remains. Of course you could plop another alias down for the sourcing, but that’s kind of gross. Instead, you can make your eb alias source after the edit is complete:

alias eb="vim ~/.bash_profile && source ~/.bash_profile"

When you use that alias, you pop into ~/bash_profile, save your change, and continue to hack away.

Using ssh bastion hosts without going mad

Fri, 20 Feb 2015 00:00:00 +0000

My employer uses ssh bastion hosts. This means in order to ssh into any server, I must first connect to a known secure server, or ‘bastion host’.

To deal with the double-connection pain, most people I know maintain a long list aliases to the servers they most commonly ssh into. For years I too maintained one of these wild configurations in my dotfiles, changing and committing it every time we stood up new servers. No more!

SSH config

My .ssh/config now contains these four precious lines:

Host bastion
  HostName some-bastion-host.umn.edu

Host *.umn.edu
  ProxyCommand ssh bastion nc %h %p

This configuration will match any host under umn.edu and transparently proxy it through the bastion host using netcat.

Bash completion

I also use bash-completion on my Mac (brew install bash-completion). This allows me to auto-complete hostnames that are in my ~/.ssh/known_hosts. With bash-completion available, ssh blah+TAB auto-completes to ssh blah.long-department-name.random-number.umn.edu.

So in the past, my ssh aliases existed for two reasons: to perform the proxy, and to shorten crazy-long hostnames. This ssh config and bash-completion make the need to maintain aliases disappear.

In search of better PeopleSoft effective date queries, Part 2

Mon, 01 Dec 2014 00:00:00 +0000

In my previous post, I made the argument that this:

SELECT ps_acad_prog.*
FROM ps_acad_prog
  JOIN (
    SELECT ps_acad_prog.emplid, ps_acad_prog.acad_career, ps_acad_prog.stdnt_car_nbr, ps_acad_prog.effdt, max(ps_acad_prog.effseq) as effseq
    FROM ps_acad_prog
      JOIN (
        SELECT emplid, acad_career, stdnt_car_nbr, max(effdt) as effdt
        FROM ps_acad_prog
        WHERE effdt <= sysdate
        GROUP BY emplid, acad_career, stdnt_car_nbr
      ) max_effdt_ps_acad_prog
        ON ps_acad_prog.emplid = max_effdt_ps_acad_prog.emplid
          AND ps_acad_prog.acad_career = max_effdt_ps_acad_prog.acad_career
          AND ps_acad_prog.stdnt_car_nbr = max_effdt_ps_acad_prog.stdnt_car_nbr
          AND ps_acad_prog.effdt = max_effdt_ps_acad_prog.effdt
    GROUP BY ps_acad_prog.emplid, ps_acad_prog.acad_career, ps_acad_prog.stdnt_car_nbr, ps_acad_prog.effdt
  ) effdt_and_effseq_acad_prog
    ON ps_acad_prog.emplid = effdt_and_effseq_acad_prog.emplid
      AND ps_acad_prog.acad_career = effdt_and_effseq_acad_prog.acad_career
      AND ps_acad_prog.stdnt_car_nbr = effdt_and_effseq_acad_prog.stdnt_car_nbr
      AND ps_acad_prog.effdt = effdt_and_effseq_acad_prog.effdt
      AND ps_acad_prog.effseq = effdt_and_effseq_acad_prog.effseq

is better than this:

SELECT *
FROM ps_acad_prog p
WHERE effdt = (
    SELECT max(effdt)
    FROM ps_acad_prog
    WHERE emplid = p.emplid
      AND acad_career = p.acad_career
      AND stdnt_car_nbr = p.stdnt_car_nbr
      AND effdt <= sysdate
  )
  AND effseq = (
    SELECT max(effseq)
    FROM ps_acad_prog
    WHERE emplid = p.emplid
      AND acad_career = p.acad_career
      AND stdnt_car_nbr = p.stdnt_car_nbr
      AND effdt = p.effdt
  )

But that first query is way bigger. Other than a speed boost, it may not clear why I think the first query is better, so I thought I’d dig into it a bit more. I’d like to explain more about why I think this is better (not just faster) and offer up some alternative notation.

Readability/Reasonability

I think the biggest reason to avoid correlated subqueries is not the performance benefit, it’s the cognitive load that it forces upon us. A big, gnarly query with many correlated subqueries makes it difficult to reason about.

When you encounter a big query that’s doing more than you can take in at once, what’s the first thing you do? You break it down. Start executing pieces of the query to get an idea of the relations that are at play in your mind. Correlated subqueries make that difficult because you cannot select a piece of the query and run it individually. Especially in the case of PeopleSoft, these subqueries are wired to other relations outside of the immediate scope.

Effective date subqueries also often get thrown into the WHERE clause alongside conditions that represent real business logic, clouding the intention of the query. The goal should be to shove all of this accidental complexity aside, leaving the minimal amount of code to represent the core of what the query is trying to accomplish.

The following query speaks for itself, doesn’t it? This should be our goal. Any effective dated nonsense that this requires should take a back seat to this distilled statement.

SELECT * FROM effective_acad_prog_tbl WHERE acad_prog='123ASDF'

Destructuring with `WITH`

Since we’re specifically talking PeopleSoft/Oracle here, we have the often neglected WITH clause at our disposal. We can use WITH to destructure the ugly peripheral bits into meaningful relation names.

In Clojure and other lisps, destructuring is done with the let form:

(let [acad-prog-eff-keys (...)
      effective-acad-prog (...)]
  (do-something effective-acad-prog))

Oracle’s WITH clause is very similar:

WITH acad_prog_tbl_eff_keys AS (
  SELECT institution, acad_prog, max(effdt) as effdt
  FROM ps_acad_prog_tbl
  WHERE effdt <= sysdate
  GROUP BY institution, acad_prog
),
effective_acad_prog_tbl AS (
  SELECT ps_acad_prog_tbl.*
  FROM ps_acad_prog_tbl
    JOIN acad_prog_tbl_eff_keys
      ON ps_acad_prog_tbl.institution = ps_acad_prog_tbl_eff_keys.institution
        AND ps_acad_prog_tbl.acad_prog = ps_acad_prog_tbl_eff_keys.acad_prog
        AND ps_acad_prog_tbl.effdt = ps_acad_prog_tbl_eff_keys.effdt
)
SELECT * FROM effective_acad_prog_tbl WHERE acad_prog = '123ASDF'

WITH lets us nicely break up (and push aside) the non-core queries and clauses, leaving the business logic to speak for itself.

Composability

So my point here is really composability. Regardless of whether you’re using WITH clauses, building queries by composing discrete meaningfully-named relations will help build clearer, faster, and more maintainable queries.

chrisdinger.com

Controlling my garage door with Elixir and Nerves

Hardware

Software

Firmware

Activating the opener

Tying it all together

RADIUS authentication in SQL Developer on MacOS (updated)

1. Install instantclient

2. sqlnet.ora

3. Mangle the java.library.path property

4. Set the TNS_ADMIN path for GUI applications

4. Configure SQL Developer to use OCI

5. Verify

References

How I learned to stop whining and tolerate my touchbar MacBook Pro

RADIUS authentication in SQL Developer on MacOS

A note on versions

1. Install instantclient

2. sqlnet.ora

3. Mangle the java.library.path property

4. Set the TNS_ADMIN path for GUI applications

4. Configure SQL Developer to use OCI

5. Verify

References

Postgres foreign data wrapper for Oracle (oracle_fdw) on MacOS

1. Install Postgres

2. Install instantclient

3. Build oracle_fdw

Option 1: alter the oracle_fdw binary

Option 2: symlink libclntsh.dylib.11.1 to an expected location

4. Install oracle_fdw

Using erloci to talk to Oracle in Elixir

Connect

Select

Insert

Errors

The Roman Numerals kata

Special cases in disguise

Make everything the same, functionally

Self-sourcing bash profile

Using ssh bastion hosts without going mad

SSH config

Bash completion

In search of better PeopleSoft effective date queries, Part 2

Readability/Reasonability

Destructuring with WITH

Composability

Destructuring with `WITH`