Celebrate your Annual `Octocat Day`?

Adorable website throws confetti in celebration of your GitHub join date:
Useful, since after 30 days join date is difficult to find

Octocat Birthday celebration site unsurprisingly located on… wait for it…
GitHub Pages: https://nomangul.github.io/octocat-day/

I thought this was so cute, I thought I’d put it up real quick. I found it while wondering in earnest when I had signed up for GitHub. Turns out it was a lot earlier than I remembered – not that I’ve been active the entire time.

I found it rather unusual that, while other platforms like Twitter gleefully display your join date on your profile page, GitHub not only obscures the date you joined after 30 days of membership, but Google is awash with discussions about how to obscure its visibility even before 30 days is over.

One example of many: https://stackoverflow.com/questions/66988864/is-it-possible-to-hide-joined-date-of-github

What’s wrong? Are people afraid of looking … uncommitted? (womp womp)

It’s a little ridiculous, since version control activity was never meant to be a competition, but I understand perhaps dates might not align with embelleshments (intended or inadvertant) made while seeking employment. Not casting aspersions, since I couldn’t remember when I joined, either!

Maybe if GitHub were more forthcoming about user join date, it’d be easier to provide accurate information. But, thanks to Octocat Day tool coming to the rescue, users can be reminded of when they joined the site with a celebration replete with confetti. πŸ₯³

Thanks to @Nomangul for making this one.

Repo: https://github.com/nomangul/octocat-day/

Have I really not written about `rsync`?

Kyun-Chan, a shy, Japanese pika disguised as a deer
Kyun-Chan, a shy, Japanese pika disguised as a deer

There’s lots of great backup tools out there – borg, rdiffbackup, bareos, zfs and btrfs send/receive, pvesync, etc. and the cutest mascotted backup program ever, of course, pikabackup (it’s adorable!) all with their own traits and best-practice use cases.

I have to say, though, call me DIY, a glutton for punishment, or just plain nerdy, but I really like my hand-written backup scripts more than anything else. Part of it is because I want to know what is happening in the process intimately enough that debugging shouldn’t be a problem, but I also find something satisfying about going through the process of identifying what each flag will do and curating them carefully for a specific use case, and (of course) learning new things about how some of my favorite timeless classics.

rsync is definitely one of those timeless classics. It’s to copying files what ssh is to remote login: Simultaneously beautiful and indispensable. And so adaptable to whatever file-level copy procedure you want to complete. For example, check out what the Arch wiki suggests for replacing your typical copy (cp) and move (mv):

# If you're not familiar, these are bash functions
# source: https://wiki.archlinux.org/title/rsync
# guessing cpr is to denote 'cp w/ rsync'

cpr() {
rsync --archive -hh --partial --info=stats1,progress2 --modify-window=1 "$@"

# these are neat because they're convenient and quiet

mvr() {
rsync --archive -hh --partial --info=stats1,progress2 --modify-window=1 --remove-source-files "$@"

It seems rsync flags are pretty personal, if someone’s familiar, usually they’ll have some favorite flags – whether they’re easy to remember, they saw them somewhere else, or think they look cool to type. I know for me, mine are -avhP for home (user files) and -aAvX for root (system files), but I painstakingly researched the documentation for this next script to create my own systemd.timer backups to (you guessed it) rsync.net:

TIMESTAMP="$(date +%Y%m%d_%Hh%Mm)"
EXCLUDE_LIST={"*.iso","*.ISO","*.img",".asdf","build",".cache",".cargo",".config/google-chrome/Default/Service\ Worker/CacheStorage",".conda","containers",".cpan",".docker","Downloads",".dotnet",".electron-gyp","grive","go",".java",".local/share/flatpak",".local/share/Trash",".npm",".nuget","OneDrive",".pnpm-store",".pyenv",".rustup",".rye",".ssh",".var","Videos","vms",".yarn"}

rsync --log-file=$LOGFILE \
    -AcDgHlmoprtuXvvvz \
    --ignore-existing \
    --fsync --delete-after \
    --info=stats3,name0,progress2 \
    --write-batch=$BATCHFILE \
    --exclude=$EXCLUDE_LIST \
    $HOME $RSYNCNET:$(hostname); \
    ssh RSYNCNET cp $LOGFILE $(hostname)/.

where $LOGFILE is the (very detailed) log of the backup, $TIMESTAMP is the time the script is invoked, and the $EXCLUDE_LIST is stuff I don’t want in my backups, like folders from other cloud services, browser cache, $HOME/.ssh, development libraries, flatpaks, and build directories for AUR and git repos.

A quick note about --exclude, it can be a little fiddly. It can be repeated for one flag at a time without an equals sign, e.g. --exclude *.iso --exclude ~/Videos, but if you want to chain them together, then they need the equals sign. I had them working from a separate file once years ago with each pattern on separate lines, but now I can’t remember how I did it, so the big one-line mess with curly braces, commas, and single-pattern quotes is how I’ve been rocking it lately. It’s ugly, but it works.

Here’s the manual for rsync in case you actually want to know what the flags are doing (definitely recommend it): https://linux.die.net/man/1/rsync

And another quick, but good, rsync reference by Will Haley: https://www.willhaley.com/blog/rsync-filters/ – this guy does all sorts of interesting stuff, and I liked his granular, yet opinionated, walk-through of rsync: how he sees it. (spoiler: two people’s rsyncs are rarely the same)

Of course, file-level backups are not the same as system images, and for that I use fsarchiver. If you’re not familiar, I definitely recommend checking them, and their awesome Arch-Based rescue ISO distro out: https://www.system-rescue.org/

I wrote my own script for that, too (of course), but I’ll probably link it in a repo since it’s quite a bit longer than the script for rsync. It is timed to run right before the rsync backup, in the same script, along with dumps of separate lists of my supported dist (pacman -Qqe) and AUR (pacman -Qqm) packages.

Oh, and also, if you ever need to do file recovery, check out granddaddy testdisk: https://www.cgsecurity.org/testdisk_doc/presentation.html

What’s your favorite backup software, and why? Any stories about how they got you (or failed to get you) out of a bind? Unfortunately, everybody’s got one these days… would love to hear about them in the comments below…

Benefactors, Meet Cartography: Using Public Disclosure Data for a Geospatial Graph with Python, Pandas, GeoPandas and Matplotlib

I was looking at datasets on data.wa.gov to see what might be fun for a project, and I came across public info disclosing the amount paid to employ WA state lobbyists. It’s a very localized representation of interests attempting to influence politics and policy for our residents and lawmakers, as it represents money spent only inside Washington State.

When I first peeked at the top of the tables, I saw disclosures from “ADVANCE CHECK CASHING” in Arlington, Virginia. I didn’t expect it at first, but I began noticing quite a few other firms lobbying us are also located outside our state. Since the level to which out-of-state firms are lobbying us isn’t something I think I’ve ever heard discussed, I became more curious about what story I could illuminate through visualizing the numbers.

If you’d like to see the source code, I created a repo on GitHub: https://github.com/averyfreeman/Python_Geospatial_Data/tree/main

The charts are generated with Matplotlib, which is a popular attempt to emulate MATLAB, the uber-capable and even more expensive software with technology previously accessible only by government, engineering firms, multinational corporations, and the ultra-wealthy. The data manipulation library is called Pandas, which is somewhat analogous to Excel, but hard to imagine when considering there’s no pointing or clicking, since there’s no user interface: All the calculations are constructed 100% with code.

This system actually has its benefits: Try loading a 35,000,000 row spreadsheet and you’re likely to have a bad time – the number of observations you can manipulate in Python dwarfs the capabilities of a spreadsheet by an insurmountable margin. I even tried to load the 12,500 row dataset for this project into Libreoffice, and the first calculation I attempted crashed immediately. And additionally, even though there’s a steeper learning curve than Excel, once people get the hang of using Pandas they can be more productive, since it’s not bogged down by all that pointing and clicking.

The data I used comes from the Public Disclosure Commission, and an explanation of the things people report are listed here: https://www.pdc.wa.gov/political-disclosure-reporting-data/open-data/dataset/Lobbyist-Employers-Summary

There’s too many columns in the disclosure data to meaningful concentrate on the money coming from each state, so I immediately narrow it down to three columns. Here, you can also see that it’s a simple .csv file being read into Python:

def geospatial_map():

    our_cols = {
        'State': 'category',
        'Year': 'int',
        'Money': 'float',

    clist = []
    for col in our_cols:

    df = pd.read_csv(csvfile, dtype=our_cols, usecols=clist)

That comes out looking like this:


   Year                 State      Money
0  2023  District of Columbia 497,305.43
1  2023            California 496,301.52
2  2022            California 446,805.56
3  2022  District of Columbia 437,504.86
4  2021            California 430,493.96

I haven’t tried it yet, but the dataset website, data.wa.gov, and all the sister-sites that are hosted on the same platform (basically every state, and the US government) have an API with a ton of SQL-like functions you can use while requesting the data. This is an amazingly powerful tool I am definitely going to try for my next project. You can read about their query functions here, if you’re curious: https://dev.socrata.com/docs/functions/#,

I took a pretty manual route and created my own data-narrowing and typing optimizer script, so the form it was in once I started making the charts was pretty different than its original state, but it helped to segment the process so I could focus on the visualization more than anything else once I got it to this point.

I still had to make sure the numbers columns didn’t have any non-numerical fields, though, otherwise they’d throw errors when trying to do any calculations. To avoid that pitfall, I filled them with zeros and made sure they were a datatype that would trip me up, either:

    df.rename(columns=to_rename, inplace=True)
    df['State'] = df['State'].fillna('Washington').astype('category')
    df['Year'] = df['Year'].fillna(0).astype(int)
    df['Money'] = df['Money'].fillna(0).astype(float)

Then, I wanted to make sure the states were organized by both state name and year, so I pivoted the table so states names were the index, and a column was created for each row. That single action aggregated all the money spent from each state by year, so at that point I had a single row per state name and a column for each year.

    # pivoting table aggregates values by year
    dfp = df.pivot_table(index='State', columns='Year', values='Money', observed=False, aggfunc='sum')
    # pivot creates yet more NaN - the following avoids peril
    dfp = dfp.fillna(value=0).astype(float) 

    # some back-of-the-napkin calculations for going forward
    first_yr = dfp.columns[0]            #    2016
    last_yr = dfp.columns[-1]            #    2023
    total_mean = dfp.mean().mean()       # 391,133
    total_median = dfp.median().median() # 141,594

It’s easier to see it than imagine what it’d look like (the dfp rather than df variable name I created to denote df pivoted):

# here's what the pivoted table looks like:

Year               2016         2017         2018  ...         2021         2022         2023
State                                              ...                                       
Alabama            0.00         0.00         0.00  ...         0.00         0.00       562.50
Arizona      156,000.00   192,778.93   231,500.00  ...   264,334.00   170,300.00   205,250.00
Arkansas      94,924.50   128,594.00   121,094.00  ...   120,501.00   104,968.84   103,384.62
California 2,606,222.26 3,232,131.73 3,751,648.42  ... 5,261,021.97 5,491,396.87 6,200,283.10
Colorado     215,818.82   195,463.67   192,221.84  ...   233,031.86   289,434.81   157,109.81

Now, on to the mapping. There’s GeoJSON data on the same web site, which can deliver similar results to a shape file (in theory), but for this first project I made use of the geospatial boundary maps available from the US Census Bureau – they have a ton of neat maps located here: https://www.census.gov/geographies/mapping-files/time-series/geo/carto-boundary-file.html (I’m using cb_2018_us_state_500k )

    shape = gpd.read_file(shapefile)
    shape = pd.merge(

The shape file is basically just like a dataframe, it just has geographic coordinates that allow Python to use for drawing boundaries. Conveniently, the State column from the dataset, and NAME column from the shape file had the same values, so I used them to merge the two together.

         NAME LSAD         ALAND       AWATER  ...         2021         2022         2023  8 year total
0     Alabama   00  131174048583   4593327154  ...         0.00         0.00       562.50        562.50
1     Arizona   00  294198551143   1027337603  ...   264,334.00   170,300.00   205,250.00  1,818,537.93
2    Arkansas   00  134768872727   2962859592  ...   120,501.00   104,968.84   103,384.62  1,006,918.19
3  California   00  403503931312  20463871877  ... 5,261,021.97 5,491,396.87 6,200,283.10 37,839,827.01
4    Colorado   00  268422891711   1181621593  ...   233,031.86   289,434.81   157,109.81  1,961,231.65

Then I animated each year’s dollar figures a frame at a time, with the 8-year aggregate calculated at the end, with the scale remaining the same to produce our dramatic finale (it’s a little contrived, but I thought it’d be fun).

At this point in the code base, it starts going from pandas (the numbers) and geopandas (the geospatial boundaries) to matplotlib (the charts/graphs/figures), and that’s where the syntax takes a big turn from familiar Python to imitation MATLAB. And it takes a bit to get used to, since it essentially has no other analog I’m familiar with (feel free to correct me in the comments below)

And it’s a very capable, powerful, language, that also happens to be quite fiddly, IMO. For example, this might sound ridiculous to anyone except people who’ve done this before, but these 4 lines are just for the legend at the bottom, with the comma_fmt line being solely responsible for commas and dollar signs (I’m not joking).

    norm = plt.Normalize(vmin=all_cols_min, vmax=upper_bounds)
    comma_fmt = FuncFormatter(lambda x, _: f'${round(x, -3):,.0f}')

    sm = plt.cm.ScalarMappable(cmap='RdBu_r', norm=norm)
    sm.set_array([])  # Only needed for adding the colorbar
    colorbar = fig.colorbar(sm, ax=ax, orientation='horizontal', shrink=0.7, format=comma_fmt)

But it’s all entertaining, nonetheless. I also did a horizontal bar-chart animation with the same data, so the dollar figures would be clear (the data from these charts perfectly correlate):

Most other states don’t have that much of an interest in Washington State politics, but there’s enough who do for it to make it interesting to see where the money is coming from. Although, I have to say, when I first charted this journey by laying eyes on a check cashing / payday loan company, it wasn’t a huge surprise.

Some other interesting info has more to do with spending by entities located inside Washington state – and to be clear, spending from inside the state far outpaces spending from outside, for obvious reasons. These two graphs are part of a work in progress. They’re derived from same dataset, but with a slightly different focus: Instead of organizing these funding sources by location, they focus on names and display exactly who is hiring the lobbyists, and, perhaps more importantly, for how for much. For example, here’s the top 10 funders of lobbyists in WA by aggregate spending:

It’s been a really fun project, and I hope I have opportunities to make more of them going forward. So many things in our world are driven by data, but the numbers often don’t speak for themselves, and our ability to tell stories with data and highlight certain issues is a necessity in conveying the importance of so many salient issues of our time.

I’ll definitely be adding more as I finish other charts, and demonstrations through a jupyter notebook. I am also really anxious to try connecting the regularly updated data through the API, which I already have access to, I just have to wire up the request client – easier said than done, but I’ve been able to do it before, so I am confident I’ll be seeing more of you soon. Thanks for visiting!

Arch Linux for Windows: Now available from the Microsoft Store

Looks like I’m not the only one who noticed

Does anyone else sense the irony?

Yes, Arch Linux is available from the Microsoft Store, and no, I’m not kidding. Go ahead and see for yourself: https://apps.microsoft.com/detail/9mznmnksm73x?hl=en-us&gl=US

My immediate reaction when I saw this was, “woah, really?”, “that’s crazy”, and, “I never thought I’d see this day”, and I wonder if it’s as jarring to other people who haven’t grown up with Microsoft being such a behemoth in the news and in their lives.

I think my dad might have joked once, “if I had bought shares in Microsoft when you were a kid, instead of your Speak n’ Spell, we’d both be retired by now.” I remember being horrified by the suggestion, too. “No, dad! Really? My Speak n’ Spell?”

Most people saw over the years that Microsoft’s success was less of a testament to their ability to innovate, but to engage in monopolistic business practices. And it was certainly effective: Microsoft was ubiquitous, and everyday people across the country fantasized about where they might be had they bought stock in MS while they could afford it.

Having grown up hearing about them choking the competition all my life is certainly one of many reasons seeing Arch Linux in the Microsoft store is really super shocking. 😳

What’s next, Microsoft open sourcing their technologies?


Back in the late ’90s / early ‘2000s, when the internet was still relatively new, Friendster was a thing, and RedHat was the only distro I’d ever heard of by name, there was a palpable sense that nothing would ever disrupt the Windows/MacOS duopoly, or cause them to reexamine their ultra-competitive business models.

The closed-intellectual property business model of Microsoft (and even more rabidly authoritarian monopolistic single-hardware vendor to walled-garden ecosystem Apple) seemed like one constant that would change about as much as death or taxes. Nobody expected Microsoft would start to give up its iron grip on source code, or start to even publish and market its own services using a Linux Operating System. (Yes, I’m looking at you, Azure hosted Kubernetes)

Bill Gates was still CEO, and they’d just had a hugely consequential precedent-setting battle in the courtroom for breaking anti-trust laws.

Microsoft bought or suffocated any decent competition, and leveraged their market proliferation into monopolization wherever it could. They eventually lost their lawsuit after several years of defending their position to achieve defacto omnipotence. It was surprising to see such a large, powerful company actually lose in court, but many saw the loss as a testament to how many bad faith business practices MS had engaged in. You have to be pretty anti-competition in this day and age to get charged by the federal government.

With a reputation like they had at the time, Anyone who knew anything about Microsoft was convinced they’d be going the locked-down software-selling, IP hoarding route forever. And while they’re certainly not perfect, they’re certainly engaging in practices I’ve never expected from them. Not in a million years.

The Microsoft I remember would have attempted to make it harder to run other company’s software. They’d never dream of providing Kubernetes on Azure with Linux containers, they’d make them all run Windows at $149 a pop, and force everyone to keep a copy of “MS Azure Browser” on their desktop.

And while I applaud MS for not pushing the boundaries of price elasticity as much as most other megacorps these days (i.e. Windows is still “only” $149) Linux’s source code is distributed for free in a way everyone can examine, so it’s hard to see how they could use their previously archetypal hoard-and-license IP business strategy.

What’s going on here? Are things actually changing?

To be clear, I know this distribution of Arch Linux for WSL has nothing to do with Microsoft really, other than it’s packaged for their proprietary VM layer on their still-$149-a-license operating system. But how jarring it is for me personally to see Microsoft change in this manner is pretty hard to overstate. I know MS is still making boatloads of money, but it’s amazing that something, or someone, could shake up the paradigm of how Microsoft Makes Money so fundamentally. And whether that was their intention, or it is simplyan artifact, there’s one person I believe we can point to for this eventually occurring (albeit, over the course of about 30-40 years):

Richard Stallman is the creator of the GNU Public License. Seen here playing your friend’s hippy dad taking you both to a cookout, he is the original stalwart advocate for free software distribution.

Stallman and his brainchild, the GPL, have been making the case that developers (and software companies) shouldn’t just release their code so people can know what it’s doing, but if they adopt any code released under GPL they are essentially forced to. Many other types of free distribution licenses allow companies to eventually close their source code and hoard their IP. There are plenty of reasons to do that, like security, competition, and market share. And if I understand correctly, it was an essential reason companies like Nintendo and Apple chose BSD for their OS platforms instead of Linux.

This is not meant to be a bash against companies for being greedy, under capitalism it’s essentially their main job. But it’s possible to be altruistic and make money. I don’t have any empirical evidence to support this suggestion, but I imagine enough developers realized hoarding IP is a barrier to improving technology, and whether for altruism, concerns about long-term self-preservation, or because their program relies heavily on a gzip library, enough people have gotten onboard with open-sourcing technology that it’s even been championed by Microsoft.

I have to pause a second to take that in. Microsoft has been the poster child of IP hoarding since their inception in 1980. By no means was adopting a license that would require developers and software companies to release their source code a fait accompli, but I suppose enough developers understood how much more rapidly we would progress collectively if we all shared information about how to do things. And for the rest of us who don’t make a habit of examining questions of existential importance on a daily basis, we probably just fell into it by proximity and convenience.

But realistically speaking, it also gives me pause to think of how much progress me might not have made if it weren’t for the open source community (regardless of license). What might not have been. Where we’d still be today. If, by 2024, open sourcing hadn’t proliferated so profoundly, we wouldn’t still be searching Alta Vista, trying to make out HTML tables on our flip phones.

And I say this gratefully, and without any sense of irony, that, for one reason or another, we all have Richard Stallman to thank for releasing Arch Linux in the Microsoft Store. Thank you, Richard!

And also probably because of all the parents who had the presence of mind to buy their kids Speak n’ Spells.

Using `grep` to focus only on that which is important

How can grep clear the clutter obstructing our goals?

This is a little beginner command-line demonstration for finding the results you need most. A lot of the time in the command line, a simple ls can return way more than any one person can reasonably deal with. Most people know how to use ls with extension flags: ls *.sh ; ls -a ; ls -a .??* , but what about situations when file extensions aren’t relevant, or when doing other things?

For those who don’t already know, grep is your spectacularly special search superhero you can use to slice and dice strings in a smattering of situations – and here’s a few instructive ways it can be used to make life easier when looking for something.

Say I’m troubleshooting kernel-install. kernel-install is an intrinsic part of making sure kernels are installed properly, but on most (if not all) distros, it’s packaged with systemd, a fairly sprawling package with tentacles in virtually everything. How can we sift through the systemd package to narrow down what we’re looking for, so our search becomes less overwhelming?

❯ pacman -Ql systemd | wc -l

A quick analysis of the number of lines contained in the systemd package is 1,450 lines! That’s a lot of stuff to sort through in order to find a rather narrow section of it we need for our issue. That’s where some strategic use of the grep command can come in handy:

❯ pacman -Ql systemd | grep -v -E 'zsh|bash|man|polkit|locale' | wc -l

I usually start by thinking about what I don’t want in the results, since there’s almost always a lot more of what we don’t need than what we do need.

This filter with grep -v does the opposite of what grep does by default, and removes everything in a given search string. By using -E we can chain strings together, as if to instruct grep not to return results with any of these values (note the single quotes and pipe character separators)

If we use grep to omit all zsh functions, bash-completions, man files, polkit definitions, and locale settings, that can get us a little closer, but it’s still only narrowed down by 577 results. systemd is a lengthy package, indeed.

❯ pacman -Ql systemd | grep -v -E 'zsh|bash|man|polkit|locale' | grep kernel
systemd /etc/kernel/
systemd /etc/kernel/install.d/
systemd /usr/bin/kernel-install
systemd /usr/lib/kernel/
systemd /usr/lib/kernel/install.conf
systemd /usr/lib/kernel/install.d/
systemd /usr/lib/kernel/install.d/50-depmod.install
systemd /usr/lib/kernel/install.d/90-loaderentry.install
systemd /usr/lib/kernel/install.d/90-uki-copy.install
systemd /usr/lib/systemd/system/sockets.target.wants/systemd-udevd-kernel.socket
systemd /usr/lib/systemd/system/sys-kernel-config.mount
systemd /usr/lib/systemd/system/sys-kernel-debug.mount
systemd /usr/lib/systemd/system/sys-kernel-tracing.mount
systemd /usr/lib/systemd/system/sysinit.target.wants/sys-kernel-config.mount
systemd /usr/lib/systemd/system/sysinit.target.wants/sys-kernel-debug.mount
systemd /usr/lib/systemd/system/sysinit.target.wants/sys-kernel-tracing.mount
systemd /usr/lib/systemd/system/systemd-udevd-kernel.socket

❯ pacman -Ql systemd | grep -v -E 'zsh|bash|man|polkit|locale' | grep kernel | wc -l

From there, I realize we got lucky looking for kernel-install, since anything related to it is likely to have the word kernel in it. A quick addition of grep kernel as a subsequent filter on the end narrowed our search down to 17 results.

Now the list is fairly manageable. But what if we want it to be very accurate?

❯ pacman -Ql systemd | grep -v -E 'zsh|bash|man|polkit|locale|mount|socket' | grep kernel
systemd /etc/kernel/
systemd /etc/kernel/install.d/
systemd /usr/bin/kernel-install
systemd /usr/lib/kernel/
systemd /usr/lib/kernel/install.conf
systemd /usr/lib/kernel/install.d/
systemd /usr/lib/kernel/install.d/50-depmod.install
systemd /usr/lib/kernel/install.d/90-loaderentry.install
systemd /usr/lib/kernel/install.d/90-uki-copy.install

❯ pacman -Ql systemd | grep -v -E 'zsh|bash|man|polkit|locale|mount|socket' | grep kernel | wc -l

At this point, I’d go back to considering what not to include – in this case I know we don’t want any mount or socket files.

Are you noticing a pattern in these results, though? They include the files we’re looking for, related to kernel-install infrastructure, but also the folders. We know which folders they’re in through getting results for the files, so the folder locations are redundant:

❯ pacman -Ql systemd | grep -v -E 'zsh|bash|man|polkit|locale|mount|socket|/$' | grep kernel
systemd /usr/bin/kernel-install
systemd /usr/lib/kernel/install.conf
systemd /usr/lib/kernel/install.d/50-depmod.install
systemd /usr/lib/kernel/install.d/90-loaderentry.install
systemd /usr/lib/kernel/install.d/90-uki-copy.install

❯ pacman -Ql systemd | grep -v -E 'zsh|bash|man|polkit|locale|mount|socket|/$' | grep kernel | wc -l

This handy filter, grep -v '/$' should remove all the lines that end with a forward slash: / — and look, it ends up being only 5 related files: kernel-install, the binary, and the included .install scripts.

Since kernel is a fairly unique word, not often used in more than one context, it’s one of the easier ones to narrow down. We could have started by looking for the word kernel first:

❯ pacman -Ql systemd | grep 'kernel' | grep -v -E '/$'
systemd /usr/bin/kernel-install
systemd /usr/lib/kernel/install.conf
systemd /usr/lib/kernel/install.d/50-depmod.install
systemd /usr/lib/kernel/install.d/90-loaderentry.install
systemd /usr/lib/kernel/install.d/90-uki-copy.install
systemd /usr/lib/systemd/system/sockets.target.wants/systemd-udevd-kernel.socket
systemd /usr/lib/systemd/system/sys-kernel-config.mount
systemd /usr/lib/systemd/system/sys-kernel-debug.mount
systemd /usr/lib/systemd/system/sys-kernel-tracing.mount
systemd /usr/lib/systemd/system/sysinit.target.wants/sys-kernel-config.mount
systemd /usr/lib/systemd/system/sysinit.target.wants/sys-kernel-debug.mount
systemd /usr/lib/systemd/system/sysinit.target.wants/sys-kernel-tracing.mount
systemd /usr/lib/systemd/system/systemd-udevd-kernel.socket
systemd /usr/share/bash-completion/completions/kernel-install
systemd /usr/share/man/man7/kernel-command-line.7.gz
systemd /usr/share/man/man8/kernel-install.8.gz
systemd /usr/share/man/man8/systemd-udevd-kernel.socket.8.gz
systemd /usr/share/zsh/site-functions/_kernel-install

❯ pacman -Ql systemd | grep 'kernel' | grep -v -E '/$'  | wc -l

Since we put grep kernel first, and did what was most obvious (filter out all the bare folder results), this search becomes admittedly less complicated. However, I thought it might be helpful to demonstrate a more narrow, selective reduction with search filters, since there are likely to be situations where one is looking for something more difficult to narrow down, and eliminating results might make more sense than immediately looking for keywords related to it.

E.g. all executable binaries and libexec files in the libvirt package:

❯ pacman -Ql libvirt | grep -v -E 'zsh|bash|man|polkit|locale|mount|socket|/$|\.'
libvirt /usr/bin/libvirtd
libvirt /usr/bin/virsh
libvirt /usr/bin/virt-admin
libvirt /usr/bin/virt-host-validate
libvirt /usr/bin/virt-login-shell
libvirt /usr/bin/virt-pki-query-dn
libvirt /usr/bin/virt-pki-validate
libvirt /usr/bin/virt-qemu-qmp-proxy
libvirt /usr/bin/virt-qemu-run
libvirt /usr/bin/virt-qemu-sev-validate
libvirt /usr/bin/virt-ssh-helper
libvirt /usr/bin/virt-xml-validate
libvirt /usr/bin/virtchd
libvirt /usr/bin/virtinterfaced
libvirt /usr/bin/virtlockd
libvirt /usr/bin/virtlogd
libvirt /usr/bin/virtlxcd
libvirt /usr/bin/virtnetworkd
libvirt /usr/bin/virtnodedevd
libvirt /usr/bin/virtnwfilterd
libvirt /usr/bin/virtproxyd
libvirt /usr/bin/virtqemud
libvirt /usr/bin/virtsecretd
libvirt /usr/bin/virtstoraged
libvirt /usr/bin/virtvboxd
libvirt /usr/lib/libvirt/libvirt_iohelper
libvirt /usr/lib/libvirt/libvirt_leaseshelper
libvirt /usr/lib/libvirt/libvirt_lxc
libvirt /usr/lib/libvirt/libvirt_parthelper
libvirt /usr/lib/libvirt/virt-login-shell-helper
libvirt /usr/share/doc/libvirt/examples/sh/virt-lxc-convert

❯ pacman -Ql libvirt | grep -v -E 'zsh|bash|man|polkit|locale|mount|socket|/$|\.' | wc -l

This is a fairly good example, because how do you search for executables? Well, they don’t usually include a file extension, so that’s exactly what we filtered out with \. at the end (the dot needs to be escaped).

And, indeed, these are essentially all executable files, even the last one in /usr/share/doc (In the case of kernel-install, the .install files are executables, so it’s an awkward juxtaposition, but that’s not very common… I’m sure you get the idea…) A couple quick, last ones, along the same vein:

❯ pacman -Ql libvirt | grep '\.sh'
libvirt /usr/lib/libvirt/libvirt-guests.sh

What shell scripts are included in a package? That’s a good one, and from our first example:

❯ pacman -Ql dracut sbctl systemd systemd-ukify | grep -i -E '\.install|\.sh' | grep -ivE 'modules.d|x11' | sort -n
dracut /usr/lib/dracut/dracut-functions.sh
dracut /usr/lib/dracut/dracut-init.sh
dracut /usr/lib/dracut/dracut-logger.sh
dracut /usr/lib/dracut/dracut-version.sh
dracut /usr/lib/kernel/install.d/50-dracut.install
dracut /usr/lib/kernel/install.d/51-dracut-rescue.install
sbctl /usr/lib/kernel/install.d/91-sbctl.install
systemd-ukify /usr/lib/kernel/install.d/60-ukify.install
systemd /usr/lib/kernel/install.d/50-depmod.install
systemd /usr/lib/kernel/install.d/90-loaderentry.install
systemd /usr/lib/kernel/install.d/90-uki-copy.install

What .install or .sh scripts are included in dracut, sbctl, systemd and systemd-ukify – sorted in numeric order (which reflects the order of execution in most freedesktop.org software). There’s a little “flag-stacking” in that grep -ivE command, too.

Anyway, I hope this demo helps people. Feel free to leave comments or suggestions, especially if there’s anything you think I missed or left out. Thanks!

Outputting a list of variables to `json` or `yaml` using `column` and `goyq`

Quick little command line kung fu job with this parser called column

 column -h

 column [options] [<file>...]

Columnate lists.

 -t, --table                      create a table
 -n, --table-name <name>          table name for JSON output
 -O, --table-order <columns>      specify order of output columns
 -C, --table-column <properties>  define column
 -N, --table-columns <names>      comma separated columns names
 -l, --table-columns-limit <num>  maximal number of input columns
 -E, --table-noextreme <columns>  don't count long text from the columns to column width
 -d, --table-noheadings           don't print header
 -m, --table-maxout               fill all available space
 -e, --table-header-repeat        repeat header for each page
 -H, --table-hide <columns>       don't print the columns
 -R, --table-right <columns>      right align text in these columns
 -T, --table-truncate <columns>   truncate text in the columns when necessary
 -W, --table-wrap <columns>       wrap text in the columns when necessary
 -L, --keep-empty-lines           don't ignore empty lines
 -J, --json                       use JSON output format for table

 -r, --tree <column>              column to use tree-like output for the table
 -i, --tree-id <column>           line ID to specify child-parent relation
 -p, --tree-parent <column>       parent to specify child-parent relation

 -c, --output-width <width>       width of output in number of characters
 -o, --output-separator <string>  columns separator for table output (default is two spaces)
 -s, --separator <string>         possible table delimiters
 -x, --fillrows                   fill rows before columns

 -h, --help                       display this help
 -V, --version                    display version

For more details see column(1).

It’s in the util-linux package, which, if I am remembering correctly, is included by default in basically any release outside of netboot and cloud images

❯ pacman -Qi util-linux
Name            : util-linux
Version         : 2.40.1-1
Description     : Miscellaneous system utilities for Linux
Architecture    : x86_64
URL             : https://github.com/util-linux/util-linux
Licenses        : BSD-2-Clause  BSD-3-Clause  BSD-4-Clause-UC  GPL-2.0-only
                  GPL-2.0-or-later  GPL-3.0-or-later  ISC  LGPL-2.1-or-later
Groups          : None
Provides        : rfkill  hardlink
Depends On      : util-linux-libs=2.40.1  coreutils  file  libmagic.so=1-64
                  glibc  libcap-ng  libxcrypt  libcrypt.so=2-64  ncurses
                  libncursesw.so=6-64  pam  readline  shadow  systemd-libs
                  libsystemd.so=0-64  libudev.so=1-64  zlib
Optional Deps   : words: default dictionary for look
Required By     : apr  arch-install-scripts  base  devtools  dracut  f2fs-tools
                  fakeroot  inxi  jfsutils  keycloak  libewf  nilfs-utils
                  ntfsprogs-ntfs3  nvme-cli  ostree  quickemu  reiserfsprogs
                  sbctl  systemd  zeromq
Optional For    : e2fsprogs  gzip  syslinux
Conflicts With  : rfkill  hardlink
Replaces        : rfkill  hardlink
Installed Size  : 14.47 MiB
Packager        : Christian Hesse <eworm@archlinux.org>
Build Date      : Mon 06 May 2024 12:22:23 PM PDT
Install Date    : Wed 08 May 2024 09:27:41 AM PDT
Install Reason  : Installed as a dependency for another package
Install Script  : No
Validated By    : Signature

this demonstrates why it’s included: It’s required by basically all of the setup infrastructure packages (makes sense). But I don’t really see column getting a lot of attention. Not sure why.

Here I used it to convert a list of installed flatpak containers to both JSON and YAML, using the column names from the flatpak’s output options (see flatpak list --help for available columns).

Start by making a list using the flatpak list --columns=$COLUMNS command:

# export comma-separated column headers, in desired order
export COLUMNS='name,application,version,branch,arch,origin,installation,ref'

# make a list of flatpaks with these columns
flatpak list --app --columns=$NAMES > all-flatpak-apps.list

# verify the list was created
cat all-flatpak-apps.list

# output (truncated):
Delta Chat	chat.delta.desktop	v1.44.1	stable	x86_64	flathub	user	chat.delta.desktop/x86_64/stable
Twilio Authy	com.authy.Authy	2.5.0	stable	x86_64	flathub	user	com.authy.Authy/x86_64/stable
Discord	com.discordapp.Discord	0.0.54	stable	x86_64	flathub	user	com.discordapp.Discord/x86_64/stable
GitButler	com.gitbutler.gitbutler	0.11.7	stable	x86_64	flathub	user	com.gitbutler.gitbutler/x86_64/stable
Drawing	com.github.maoschanz.drawing	1.0.2	stable	x86_64	flathub	user	com.github.maoschanz.drawing/x86_64/stable
Flatseal	com.github.tchx84.Flatseal	2.2.0	stable	x86_64	flathub	user	com.github.tchx84.Flatseal/x86_64/stable
Extension Manager	com.mattjakeman.ExtensionManager	0.5.1	stable	x86_64	flathub	user	com.mattjakeman.ExtensionManager/x86_64/stable
Yubico Authenticator	com.yubico.yubioath	7.0.0	stable	x86_64	flathub	user	com.yubico.yubioath/x86_64/stable
. . . 

It’s kinda hard to read like that, but I suppose if your font were small enough, or the terminal wide enough, the lines would stop wrapping and they’d be easier to read…

Then, take the list and convert it to json with a little column parsing:

# the flags are: 
# 1. create a table (-t), 
# 2. give it these headers (-N $comma,separated,headervals) 
# 3. limit it to the number we gave you (-l $integer)
# 4. make it json! (-J)
column -t -d -N $NAMES -l 8 -J all-flatpak-apps.list > flatpak-apps.json

# column is a lot more forgiving of formatting than jq and yq

I haven’t had a lot of luck piping to column directly to make a parse-inception one-liner, which is why I am making two files (the flatpak.list and the .json file). I am sure there’s some people out there who could make this work, but I want to keep it on the less complicated side for demonstrative purposes, as well.

The output will be json format with the column parse by itself, but some of us like to pretty-print our json with jq , since the colors make it easier to read – especially when they’re not even pretty-printed:

column -t -d -N $NAMES -l 8 -J all-flatpak-apps.list | jq  

  "table": [
      "name": "Delta",
      "application": "Chat",
      "version": "chat.delta.desktop",
      "branch": "v1.44.1",
      "arch": "stable",
      "origin": "x86_64",
      "installation": "flathub",
      "ref": "user\tchat.delta.desktop/x86_64/stable"
      "name": "Twilio",
      "application": "Authy",
      "version": "com.authy.Authy",
      "branch": "2.5.0",
      "arch": "stable",
      "origin": "x86_64",
      "installation": "flathub",
      "ref": "user\tcom.authy.Authy/x86_64/stable"

      "name": "UPnP",
      "application": "Router",
      "version": "Control",
      "branch": "org.upnproutercontrol.UPnPRouterControl",
      "arch": "0.3.4",
      "origin": "stable",
      "installation": "x86_64",
      "ref": "flathub\tuser\torg.upnproutercontrol.UPnPRouterControl/x86_64/stable"
      "name": "Wireshark",
      "application": "org.wireshark.Wireshark",
      "version": "4.2.5",
      "branch": "stable",
      "arch": "x86_64",
      "origin": "flathub",
      "installation": "user",
      "ref": "org.wireshark.Wireshark/x86_64/stable"
      "name": "ZAP",
      "application": "org.zaproxy.ZAP",
      "version": "2.15.0",
      "branch": "stable",
      "arch": "x86_64",
      "origin": "flathub",
      "installation": "user",
      "ref": "org.zaproxy.ZAP/x86_64/stable"

And if you need yaml, the same thing works with yq:

# what NOT to do (if you want to be able to read it):

column -t -d -N $NAMES -l 8 -J all-flatpak-apps.list | yq
{"table": [{"name": "Delta", "application": "Chat", "version": "chat.delta.desktop", "branch": "v1.44.1", "arch": "stable", "origin": "x86_64", "installation": "flathub", "ref": "user\tchat.delta.desktop/x86_64/stable"}, {"name": "Twilio", "application": "Authy", . . . 

# sometimes yq requires being super-explicit:

# yq flags are: 
# 1. input file type (-p type)
# 2. output file type (-o type)
# 3. pretty-print (-P) 
column -t -d -N $NAMES -l 8 -J all-flatpak-apps.list | yq -p json -o yaml -P
  - name: Delta
    application: Chat
    version: chat.delta.desktop
    branch: v1.44.1
    arch: stable
    origin: x86_64
    installation: flathub
    ref: "user\tchat.delta.desktop/x86_64/stable"
  - name: Twilio
    application: Authy
    version: com.authy.Authy
    branch: 2.5.0
    arch: stable
    origin: x86_64
    installation: flathub
    ref: "user\tcom.authy.Authy/x86_64/stable"
  - name: Discord
    application: com.discordapp.Discord
    version: 0.0.54
    branch: stable
    arch: x86_64
    origin: flathub
    installation: user
    ref: com.discordapp.Discord/x86_64/stable
. . . 

I’ve got the go-yq package, which has a much newer version of jq included with it, so if you’re seeing some differences in behavior or command-line arguments, that could be why – give it a shot if you’re having issues. (I do prefer the original jq, but I don’t really have a choice since it’s a dependency of devtools

# not a huge fan of gojq-bin, but go-yq is awesome...
❯ pacman -Qi go-yq
Name            : go-yq
Version         : 4.44.1-1
Description     : Portable command-line YAML processor
Architecture    : x86_64
URL             : https://github.com/mikefarah/yq
Licenses        : MIT
Groups          : None
Provides        : None
Depends On      : glibc
Optional Deps   : None
Required By     : None
Optional For    : None
Conflicts With  : yq
Replaces        : None
Installed Size  : 9.58 MiB
Packager        : Daniel M. Capella <polyzen@archlinux.org>
Build Date      : Sat 11 May 2024 08:14:21 PM PDT
Install Date    : Sat 11 May 2024 09:08:38 PM PDT
Install Reason  : Explicitly installed
Install Script  : No
Validated By    : Signature

And don’t forget, column should be able to format basically any time of list with repeating columns. I guess that’s all for now, enjoy!

ok nevermind, just one more thing: This has nothing to do with column, necessarily, but it’s all the output you can get when you choose json as your output. I’ll have to do another post about this, but it’s tangentially related.

Look at the output from kubernetes kubectl get pods -A :

kubectl get pods -A
NAMESPACE              NAME                                        READY   STATUS    RESTARTS       AGE
default                elasticsearch-coordinating-0                1/1     Running   0              91m
default                elasticsearch-coordinating-1                1/1     Running   0              91m
default                elasticsearch-data-0                        1/1     Running   0              91m
default                elasticsearch-data-1                        1/1     Running   0              91m
default                elasticsearch-ingest-0                      1/1     Running   0              91m
default                elasticsearch-ingest-1                      1/1     Running   0              91m
default                elasticsearch-master-0                      1/1     Running   0              91m
default                elasticsearch-master-1                      1/1     Running   0              91m
kube-system            coredns-7db6d8ff4d-x8nnb                    1/1     Running   1 (101m ago)   125m
kube-system            etcd-minikube                               1/1     Running   1 (101m ago)   125m
kube-system            kube-apiserver-minikube                     1/1     Running   1 (101m ago)   125m
kube-system            kube-controller-manager-minikube            1/1     Running   1 (101m ago)   125m
kube-system            kube-proxy-jbds4                            1/1     Running   1 (101m ago)   125m
kube-system            kube-scheduler-minikube                     1/1     Running   1 (101m ago)   125m
kube-system            metrics-server-c59844bb4-d8wnd              1/1     Running   1 (101m ago)   119m
kube-system            storage-provisioner                         1/1     Running   3 (101m ago)   125m
kubernetes-dashboard   dashboard-metrics-scraper-b5fc48f67-j5znq   1/1     Running   1 (101m ago)   121m
kubernetes-dashboard   kubernetes-dashboard-779776cb65-rh8s2       1/1     Running   2 (101m ago)   121m
portainer              portainer-7bf545c674-xnjjr                  1/1     Running   1 (101m ago)   119m
yakd-dashboard         yakd-dashboard-5ddbf7d777-mzqk2             1/1     Running   1 (101m ago)   119m

It’s not bad, but check out all the info you get when you add --output json:

# +`jq` to colorize the output (or `yq -p json -o json -P`)
kubectl get pods -A -o json | jq 
  "apiVersion": "v1",
  "items": [
      "apiVersion": "v1",
      "kind": "Pod",
      "metadata": {
        "creationTimestamp": "2024-05-25T12:26:56Z",
        "generateName": "elasticsearch-coordinating-",
        "labels": {
          "app": "coordinating-only",
          "app.kubernetes.io/component": "coordinating-only",
          "app.kubernetes.io/instance": "elasticsearch",
          "app.kubernetes.io/managed-by": "Helm",
          "app.kubernetes.io/name": "elasticsearch",
          "app.kubernetes.io/version": "8.13.4",
          "apps.kubernetes.io/pod-index": "0",
          "controller-revision-hash": "elasticsearch-coordinating-64759546b",
          "helm.sh/chart": "elasticsearch-21.1.0",
          "statefulset.kubernetes.io/pod-name": "elasticsearch-coordinating-0"
        "name": "elasticsearch-coordinating-0",
        "namespace": "default",
        "ownerReferences": [
            "apiVersion": "apps/v1",
            "blockOwnerDeletion": true,
            "controller": true,
            "kind": "StatefulSet",
            "name": "elasticsearch-coordinating",
            "uid": "e403f4a1-3058-4830-8f4e-25323fa68be5"
        "resourceVersion": "2737",
        "uid": "5c26ea5d-5ec7-40b1-946c-573651123552"
      "spec": {
        "affinity": {},
        "automountServiceAccountToken": false,
        "containers": [
            "env": [
                "name": "MY_POD_NAME",
                "valueFrom": {
                  "fieldRef": {
                    "apiVersion": "v1",
                    "fieldPath": "metadata.name"
                "name": "BITNAMI_DEBUG",
                "value": "false"
                "name": "ELASTICSEARCH_CLUSTER_NAME",
                "value": "elastic"
                "name": "ELASTICSEARCH_IS_DEDICATED_NODE",
                "value": "yes"
                "name": "ELASTICSEARCH_NODE_ROLES"
                "value": "9300"
                "name": "ELASTICSEARCH_HTTP_PORT_NUMBER",
                "value": "9200"
                "name": "ELASTICSEARCH_CLUSTER_HOSTS",
                "value": "elasticsearch-master-hl.default.svc.cluster.local,elasticsearch-coordinating-hl.default.svc.cluster.local,elasticsearch-data-hl.default.svc.cluster.local,elasticsearch-ingest-hl.default.svc.cluster.local,"
                "name": "ELASTICSEARCH_TOTAL_NODES",
               . . . 

It’s nice that kubectl actually has a yaml output, since it’s a little easier to read than json, what with all the [{"punctuation": "on"},{"basically": "every}, {"word": true}], but most programs, if they even offer json, probably aren’t going to offer yaml – so to convert it, here’s the pipe string:

# adding `yq` colorizes output to `yaml` output, too
kubectl get pods -A -o json | yq -p json -o yaml -P  
apiVersion: v1
  - apiVersion: v1
    kind: Pod
      creationTimestamp: "2024-05-25T12:26:56Z"
      generateName: elasticsearch-coordinating-
        app: coordinating-only
        app.kubernetes.io/component: coordinating-only
        app.kubernetes.io/instance: elasticsearch
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: elasticsearch
        app.kubernetes.io/version: 8.13.4
        apps.kubernetes.io/pod-index: "0"
        controller-revision-hash: elasticsearch-coordinating-64759546b
        helm.sh/chart: elasticsearch-21.1.0
        statefulset.kubernetes.io/pod-name: elasticsearch-coordinating-0
      name: elasticsearch-coordinating-0
      namespace: default
        - apiVersion: apps/v1
          blockOwnerDeletion: true
          controller: true
          kind: StatefulSet
          name: elasticsearch-coordinating
          uid: e403f4a1-3058-4830-8f4e-25323fa68be5
      resourceVersion: "2737"
      uid: 5c26ea5d-5ec7-40b1-946c-573651123552
      affinity: {}
      automountServiceAccountToken: false
        - env:
            - name: MY_POD_NAME
                  apiVersion: v1
                  fieldPath: metadata.name
            - name: BITNAMI_DEBUG
              value: "false"
              value: elastic
              value: "yes"
            - name: ELASTICSEARCH_NODE_ROLES
              value: "9300"
              value: "9200"
              value: elasticsearch-master-hl.default.svc.cluster.local,elasticsearch-coordinating-hl.default.svc.cluster.local,elasticsearch-data-hl.default.svc.cluster.local,elasticsearch-ingest-hl.default.svc.cluster.local,
              value: "4"
              value: elasticsearch-master-0 elasticsearch-master-1
              value: "2"
              value: $(MY_POD_NAME).elasticsearch-coordinating-hl.default.svc.cluster.local
            - name: ELASTICSEARCH_HEAP_SIZE
              value: 128m
          image: docker.io/bitnami/elasticsearch:8.13.4-debian-12-r0
          imagePullPolicy: IfNotPresent
            failureThreshold: 5
            initialDelaySeconds: 180
            periodSeconds: 10
            successThreshold: 1
              port: rest-api
            timeoutSeconds: 5
          name: elasticsearch
            - containerPort: 9200
              name: rest-api
              protocol: TCP
            - containerPort: 9300
              name: transport
              protocol: TCP
                - /opt/bitnami/scripts/elasticsearch/healthcheck.sh
            failureThreshold: 5
            initialDelaySeconds: 90
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 5
              cpu: 750m
              ephemeral-storage: 1Gi
              memory: 768Mi
              cpu: 500m
              ephemeral-storage: 50Mi
              memory: 512Mi
            allowPrivilegeEscalation: false
                - ALL
            privileged: false
            readOnlyRootFilesystem: true
            runAsGroup: 1001
            runAsNonRoot: true
            runAsUser: 1001
            seLinuxOptions: {}
              type: RuntimeDefault
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
            - mountPath: /tmp
              name: empty-dir
              subPath: tmp-dir
            - mountPath: /opt/bitnami/elasticsearch/config
              name: empty-dir
              subPath: app-conf-dir
              . . . 

It sure is nice to be able to read the stuff that comes out of the tools we’re using every day…

Using `obs` CLI controller with my new favorite mind-mapping software Obsidian – in a flatpak container

Obsidian.md mind map example

There’s always issues here or there trying to incorporate container apps with the desktop, due to their sandboxed nature and subsequent lack of out-of-the-box feature parity with their distro-packaged and supported counterparts.

This is not a post about how awesome Obsidian is, but if you haven’t heard of it, I recommend checking it out: https://obsidian.md/

Rather, it’s a response I just put on github issues for an obsidian CLI controller I have been using. It’s been pretty helpful, but the real gem is Obsidian itself, I just love it, right down to being able to export markmap files and having vim keymap settings.

I just switched to the flatpak version, though, because I noticed the package distributed through Arch’s extra repository was pulling electron on to my system as a dependency (I am trying to cordon nodejs off as much as possible to my adsf environment and away from my main system, because it’s caused the most intra-dependency issues out of any language I’ve dealt with, by far, but that’s too irritating to go into here).

The response ended up being so long, I figured I might as well make a blog post out of it. Here’s the original: https://github.com/Yakitrak/obsidian-cli/issues/20

To follow up, according to Obsidian manual on its URI interface, obs needs 3 things to interface with URI:

  1. .desktop file
  2. Exec= line pointing at executable binary
  3. %u to be a command-line argument

Separate pre-req: obs name and path should be different values

obs print-default
Default vault name:  Obsidian
Default vault path:  **$HOME/Documents/Obsidian**

The first thing I’d try is giving the included desktop file a lowercase %u

sed -i 's|%U|%u|g' $HOME/.local/share/flatpak/app/md.obsidian.Obsidian/x86_64/stable/active/export/share/applications/md.obsidian.Obsidian.desktop 

obs open your-vault-name

(md.obsidian.Obsidian should open even if completely closed)

If that doesn’t work, try these steps:

Copy the flatpak desktop file to $HOME/.local/share/applications
Name it obsidian.desktop
Point the Exec= line to flatpak executable,

Exec=$HOME/.local/share/flatpak/app/md.obsidian.Obsidian/current/active/export/bin/md.obsidian.Obsidian %u

run it with a lowercase %u which appears to be an error on both Linux package distributions I’ve tried so far

One might want to examine the wrapper included with the flatpak to see if there’s any other settings they want to incorporate for their setup if they are executing the binary directly. A lot of time these wrappers are outdated and/or unnecessary for their situation, but worth checking out: $HOME/.local/share/flatpak/app/md.obsidian.Obsidian/x86_64/stable/active/files/bin/obsidian.sh


set -oue pipefail


add_argument() {
    declare -i "$1"=${!1:-0}

    if [[ "${!1}" -eq 1 ]]; then

# Nvidia GPUs may need to disable GPU acceleration:
# flatpak override --user --env=OBSIDIAN_DISABLE_GPU=1 md.obsidian.Obsidian
add_argument OBSIDIAN_DISABLE_GPU       --disable-gpu
add_argument OBSIDIAN_ENABLE_AUTOSCROLL --enable-blink-features=MiddleClickAutoscroll

# Wayland support can be optionally enabled like so:
# flatpak override --user --socket=wayland md.obsidian.Obsidian

# Some compositors a real path a instead of a symlink for WAYLAND_DISPLAY:
# https://github.com/flathub/md.obsidian.Obsidian/issues/284
if [[ -e "${XDG_RUNTIME_DIR}/${WL_DISPLAY}" || -e "/${WL_DISPLAY}" ]]; then
    echo "Debug: Enabling Wayland backend"
    if [[ -c /dev/nvidia0 ]]; then
        echo "Debug: Detecting Nvidia GPU. disabling GPU sandbox."

# The cache files created by Electron and Mesa can become incompatible when there's an upgrade to
# either and may cause Obsidian to launch with a blank screen:
# https://github.com/flathub/md.obsidian.Obsidian/issues/214
if [[ "${OBSIDIAN_CLEAN_CACHE}" -eq 1 ]]; then
        if [[ -d "${CACHE_DIRECTORY}" ]]; then
            echo "Deleting cache directory: ${CACHE_DIRECTORY}"
            rm -rf "${CACHE_DIRECTORY}"

echo "Debug: Will run Obsidian with the following arguments: ${EXTRA_ARGS[@]}"
echo "Debug: Additionally, user gave: $@"

export FLATPAK_ID="${FLATPAK_ID:-md.obsidian.Obsidian}"

# Discord RPC
for i in {0..9}; do
    test -S "$XDG_RUNTIME_DIR"/"discord-ipc-$i" || ln -sf {app/com.discordapp.Discord,"$XDG_RUNTIME_DIR"}/"discord-ipc-$i";

zypak-wrapper /app/obsidian $@ ${EXTRA_ARGS[@]}

Also don’t forget flatseal.

`rmw` – the trash-aware `rm` your CLI probably should’ve had by default

Hasn’t everybody deleted some stuff in the command line, wishing they had a way to get it back? Well, by default there are no do-overs. That’s where rmw comes in: rmw creates a trash can for your command line, so even after you delete some files (as long as you use it), you should be able to get them back within your specified period of time.

It’s a semi-compatible drop-in replacement for rm that should leave you feeling more at home than the alternatives – I checked out at least 7 of them, and this appeared to be the most developed and supported out of all the ones I tried. And while rmw doesn’t (yet) support viewing, deleting, and otherwise co-mingling files with your desktop trash, you can at least keep them all in the same place to avoid confusion.

For me, setup went something like this …

# Set your environment variables (ephemeral and persistent):

$ export RMW_CONF_DIR="$HOME/.config/rmw"
$ echo 'export RMW_CONF_DIR="$HOME/.config/rmw"' >> $HOME/.zshrc  # or .bashrc

# Create your configuration directory and go there:

$ mkdir -p $RMW_CONF_DIR

# now this is a little weird - rmwrc has to be $CONFIG/rmwrc, but it keeps
two files for configuration in $RMW_CONF_DIR

Set up your config:

# auto-create the config with rmw:

$ rmw --config $RMW_CONF_DIR/rmwrc

The auto-created configuration has these defaults, but the Waste directory seemed a little redundant to me

$ cat rmwrc 

# rmw default waste directory, separate from the desktop trash
WASTE = $HOME/.local/share/Waste

# The directory used by the FreeDesktop.org Trash spec
# Note to macOS and Windows users: moving files to 'Desktop' trash
# doesn't work yet
# WASTE=$HOME/.local/share/Trash

# A folder can use the $UID variable.
# See the README or man page for details about using the 'removable' attribute
# WASTE=/mnt/flash/.Trash-$UID, removable

# How many days should items be allowed to stay in the waste
# directories before they are permanently deleted
# use '0' to disable purging (can be overridden by using --purge=N_DAYS)
expire_age = 0

# purge is allowed to run without the '-f' option. If you'd rather
# require the use of '-f', you may uncomment the line below.
# force_required

So I switched the commenting to use the desktop trash location instead of the new one rmw uses by default:

# Change configuration to reflect freedesktop (gnome, etc.) default trash location:

$ sed -i 's|# WASTE=$HOME/.local/share/Trash|WASTE=$HOME/.local/share/Trash|g' $RMW_CONF_DIR/rmwrc 

$ sed -i 's|WASTE = $HOME/.local/share/Waste|# WASTE = $HOME/.local/share/Waste|g' $RMW_CONF_DIR/rmwrc 

$ cat $RMW_CONF_DIR/rmwrc 
# rmw default waste directory, separate from the desktop trash
# WASTE = $HOME/.local/share/Waste

# The directory used by the FreeDesktop.org Trash spec
# Note to macOS and Windows users: moving files to 'Desktop' trash
# doesn't work yet

Also, I’d like it to empty itself after a month so I can’t forget to do it myself:

# set 30 day retention policy:

$ sed -i 's|expire_age = 0|expire_age = 30|g' $RMW_CONF_DIR/rmwrc 

$ cat $RMW_CONF_DIR/rmwrc | grep expire_age

expire_age = 30

Lastly, I’d like to try it out in place of rm for a while and see how it goes…

alias rm=rmw 

Here’s the online manual: https://theimpossibleastronaut.com/rmw-website/
And the github repo: https://github.com/theimpossibleastronaut/rmw/
Hope it gets you out of a bind!

Bring Single-Meta-Keypress Overview Behavior to KDE6 (just like Gnome!) – and a bit on KDE configuration files…

I missed KDE… best of both worlds?

Like a lot of things in the tech world these days, it all started with a short question on Reddit:

Keyboard shortcut to search programs in overview like Gnome?
byu/AveryFreeman inkde


I found out the meta-only modifier is one thing that needed to be set in $HOME/.config/kwinrc, but also the ExposeAll=Meta config in $HOME/.config/kglobalshortcutsrc – so there’s two places where it needs to be set. Also, you can add /KWin reconfigure to the end and it should start working immediately (reloads kwin config). But hey, thanks so much for pointing me in the right direction!

(my synoposis in the subred)

The biggest impediment to getting the Gnome-like super key behavior in KDE is getting KDE to interpret a single keypress as a keyboard shortcut. If you look at it critically, you can see there’s a lot going on in that single keypress in Gnome: In Gnome, a single keypress gives you both the open windows overview, and a search bar, and even has additional menus accessible from repeating the action (double-pressing super).

KDE isn’t really able to allow single-keypress shortcuts by default, but there’s a spot they’ve carved out where users can add the behavior if they really want to (I get the sense it’s caveat emptor). I’ve never been particularly interested in sticking to pre-defined conventions, so I set out to find out how we could make super behave like it does in Gnome. It couldn’t be that hard, right?

I wanted to know what was going on under the hood, so I did some digging. The most complete reference on the KDE configuration files I’ve found so far is here: https://userbase.kde.org/KDE_System_Administration/Configuration_Files

There’s a whole bunch of really great examples here: https://userbase.kde.org/Plasma/Tips

There’s some man files that are also helpful here (I don’t think this particular spec has changed from KDE 5 to 6):

I found the shortcuts in KDE6 are located in a file named /home/$USER/.config/kglobalshortcutsrc. They’re grouped by category with the names in brackets, like [ActivityManager], [kmix], etc.

So, related to my question, when I opened this file, under [kwin] I found this:

# $HOME/.config/kglobalshortcutsrc

Activate Window Demanding Attention=Meta+Ctrl+A,Meta+Ctrl+A,Activate Window Demanding Attention
Cycle Overview=Meta+Tab,none,Cycle through Overview and Grid View
Cycle Overview Opposite=none,none,Cycle through Grid View and Overview
. . . 
ExposeAll=Meta,Ctrl+F10\tLaunch (C),Toggle Present Windows (All desktops)
. . .Code language: PHP (php)

And when I ran the following:

kreadconfig6 --file kglobalshortcutsrc --group kwin --key ExposeAll

I get this response in my terminal:

. . .  
Meta,Ctrl+F10   Launch (C),Toggle Present Windows (All desktops)

So in reference to the kreadconfig6 syntax, the key is the kwin action named ExposeAll. The group should be the category of actions, [kwin]. The harder part is keeping the description in there when writing from the command line, because it’s tab-separated, \t to the right of the same line. But anyway, I am assuming it would be:

kwriteconfig6 --file kglobalshortcutsrc --group kwin --key ExposeAll Meta"

Or if you needed the descriptor to be in the same line as the value (like in the text file), it would be:

kwriteconfig6 --file kglobalshortcutsrc --group kwin --key ExposeAll Meta\tLaunch (C),Toggle Present Windows (All desktops)"

I’m not sure if kwriteconfig6 can sort out the tab-separated sections on its own, but I imagine it probably can. I have shied away from editing these config files in a text editor just in case there’s some specificity they have regarding tab and whitespace I might mess up, but I imagine it could be one option – if you try it, be sure to use a tabspace v whitespace identifier plugin for vim

Thankfully normal people can add this behavior from the GUI under Ksettings -> Shortcuts -> Kwin, too – so that’s reassuring!

A bigger issue might be specifying that you can use only a modifier key as a shortcut (e.g. meta, alt, shift, etc.) which is what the answer to my reddit question addressed (although I don’t think they were aware, exactly).

Therefore, settings need to be created or modified in two separate places, once for the key behavior, ExposeAll (previous example), and another for allowing the key to behave like a shortcut. As far as I can tell, allowing Meta on its own to ‘be’ a shortcut does need to be invoked at the command line. Here’s the process:

Meta (the keypress) is the key value in the pair, analogous to the example with ExposeAll. It will end up in the config file for kwin:

# $HOME/.config/kwinrc 

Meta=org.kde.kglobalaccel,/component/kwin,org.kde.kglobalaccel.Component,invokeShortcut,OverviewCode language: PHP (php)

Coincidentally, there the syntax to add it is provided at the end of the kwriteconfig5 man file I linked above. At the very end, it gives the example:

kwriteconfig5 --file ~/.config/kwinrc --group ModifierOnlyShortcuts --key Meta "org.kde.kglobalaccel,/component/krunner_desktop,org.kde.kglobalaccel.Component,invokeShortcut,_launch"Code language: JavaScript (javascript)

… which is for launching krunner if you hit meta by itself.

If you’d want the config to be reloaded (to start working immediately) you can add /KWin reconfigure to the end.

So I’m assuming if you already have the ExposeAll behavior in your kglobalshortcutsrc and you run the above argument, it should combine krunner with ExposeAll under the Meta keypress, which is extremely Gnome-like behavior (and pretty exciting!)

TBH I was trying all sorts of stuff when I was doing this and am not sure the exact point at which I got the desired behavior, it might require re-logging in or something like that, too. But these are the two elements you’d need in which to get the Gnome-like action from smashing your Meta key.

One last thing – you may have noticed there’s a more conventional keyboard shortcut Ctrl+F10 for ExposeAll in $HOME/.config/kglobalshortcutsrc – one that actually combines two keys, like normal (gee, seems quaint!)

. . . 
ExposeAll=Meta,Ctrl+F10\tLaunch (C),Toggle Present Windows (All desktops)
. . . 

That’s because the userbase.kde.org site I linked at the top says a single-key shortcut (Meta) might not work if there isn’t a key combination shortcut also configured for the same action (Ctrl+F10)

Lastly, I threw together a little one-liner that dumps all the configs into a single text file for analyzing. It makes them a little easier to sift and not have to open each one individually. Just run it from the $HOME/. folder:

for i in $(ls -a1 .config/k*); do echo "File: $(pwd)/$i"; echo ' '; cat $i; echo ' '; echo '--- end ---';  echo ' '; done > ./kde-config-files.txtCode language: PHP (php)

I’ll probably put all my KDE6 configs in a git repo one of these days, but for now, I’m off to study ORM development with Python and SQL… Cheers!

Name Your Distro: Flatpaks irritatingly don’t follow desktop theme (written specific to KDE / Plasma-Desktop 6)

Brings back memories of being read the Ugly Duckling

This is not an issue with any Linux distro specifically, but with all distros in general. If anyone out there’s had this issue, I’m sure you’ve shared my pain. I’d struggled with getting flatpaks to follow the (dark) system theme I set for a while, and I finally just got it fixed, so I thought I’d run through the process real quick:

  1. make sure you’re setting a theme that has a flatpak version available (I chose adw-gtk3{,-dark})
  2. install the flatpak versions: as of writing, gtk3 flatpak themes are the only ones you need to download, but if your theme doesn’t include a gtk4 variant, YMMV
  3. install the gtk themes for your desktop environment and ensure they are in the proper location
  4. enable gtk themes in gsettings
  5. A couple last-ditch effort recommendations if 1-4 doesn’t work

Here’s the steps expanded:

Choose a theme you can get in a desktop version – a good place to see if you can find the theme you want is gnome-look.org. Make sure if you use both user and system flatpaks you have a theme for both (I personally only use userspace flatpaks from flathub to avoid complications) Here’s the current list of flatpak theme runtimes as of writing:

❯ clear && flatpak search --user --columns=app theme | grep org.gtk | sort -n<br>org.gtk.Gtk3theme.Adapta<br>org.gtk.Gtk3theme.Adapta-Brila<br>org.gtk.Gtk3theme.Adapta-Brila-Eta<br>org.gtk.Gtk3theme.Adapta-Eta<br>org.gtk.Gtk3theme.Adapta-Nokto<br>org.gtk.Gtk3theme.Adapta-Nokto-Eta<br>org.gtk.Gtk3theme.Adementary<br>org.gtk.Gtk3theme.Adwaita-dark<br>org.gtk.Gtk3theme.adw-gtk3<br>org.gtk.Gtk3theme.adw-gtk3-dark<br>. . .<br>

You can see at the top of the list adw-gtk3 and adw-gtk3-dark are separate packages, so if you wanted to support both light and dark modes, install them both:

❯ for T in adw-gtk3 adw-gtk3-dark; do flatpak install -uy org.gtk.Gtk3theme.$T; done 

See if there’s a packaged or scripted version of the same theme on your distro before installing from Gnome Look – I lucked out and there was one already in the AUR (number 3 looked good to me!):

❯ paru -Ss adw-gtk3
aur/adw-gtk35.3-1 [+43 ~3.15]
Β Β Β The theme from libadwaita ported to GTK-3
aur/adw-gtk3-git1.0.r2.a2a0114-1 [+15 ~1.21]
Β Β Β The theme from libadwaita ported to GTK-3
aur/adw-gtk-theme1.1-2 [+12 ~1.00]
Β Β Β LibAdwaita Theme for all GTK3 and GTK4 Apps. NOTE: This is a meta package
   which uses adw-gtk3 for GTK3 and official LibAdwaita theme for GTK4

If you do have to install a theme package manually, just remember they can go in either /usr/share/themes/. (system-wide) or $HOME/.local/share/themes (user), but not both!

Go to kcm_style in your KDE settings panel and choose “Configure GNOME/GTK Application Style” (top right)

❯ systemsettings kcm_style
KDE Settings Configuration (part 1)
KDE Settings Configuration (part 2)

Last step (hopefully):

Tell gsettings your defaults – this has to be done from the command line unless you have dconf-editor or gnome-tweaks installed for some reason (unusual and unnecessary for a KDE-centric system). Here’s the key-value pairs:

❯ gsettings set org.gnome.desktop.interface gtk-theme 'adw-gtk3-dark'; 
  gsettings set org.gnome.desktop.interface color-scheme 'prefer-dark'

Replace the values at the end with your particular theme and preference, of course (again, I choose adw-gtk3-dark, mostly because I knew it would be the best supported theme). Obviously, if you don’t 'prefer-dark' that second line is optional.

OK now try starting one of the flatpaks you’ve been having issues with and see how it went!

If it’s still not loading in your preferred theme, I’d go back and check the steps to make sure you covered them properly. You can try gsettings get on org.gnome.desktop.interface gtk-theme to check that it has your proper value, or check /usr/share/themes to make sure a folder exists with the name of the theme you thought you installed (manually or with a package manager) or $HOME/.local/share/themes/ if you installed it manually to your user folder.

If those all seem legit, check out xsettingsd: It looks like it’s a dependency of kde-gtk-config on Arch Linux currently, so you might already have it:

❯ pacman -Qi xsettingsd
Name            : xsettingsd
Version         : 1.0.2-1
Description     : Provides settings to X11 applications via the XSETTINGS specification
Architecture    : x86_64
URL             : https://github.com/derat/xsettingsd
Licenses        : custom:BSD
Groups          : None
Provides        : None
Depends On      : libx11  gcc-libs
Optional Deps   : None
Required By     : kde-gtk-config
Optional For    : None
Conflicts With  : None
Replaces        : None
Installed Size  : 78.82 KiB
Packager        : Antonio Rojas <arojas@archlinux.org>
Build Date      : Mon 09 Aug 2021 04:16:10 AM PDT
Install Date    : Sat 16 Mar 2024 10:31:30 PM PDT
Install Reason  : Installed as a dependency for another package
Install Script  : No
Validated By    : Signature

Do the usual checks to make sure it’s running (it’s only visible using systemctl --user in Arch):

❯  systemctl --user status xsettingsd.service
● xsettingsd.service - XSETTINGS-protocol daemon
     Loaded: loaded (/usr/lib/systemd/user/xsettingsd.service; static)
     Active: active (running) since Sat 2024-03-23 14:35:18 PDT; 1h 52min ago
   Main PID: 469119 (xsettingsd)
      Tasks: 1 (limit: 9084)
     Memory: 156.0K (peak: 696.0K swap: 344.0K swap peak: 344.0K zswap: 55.8K)
        CPU: 4ms
     CGroup: /user.slice/user-1000.slice/user@1000.service/session.slice/xsettingsd.service
             └─469119 /usr/bin/xsettingsd

Mar 23 14:35:18 purplehippo systemd[1283]: Started XSETTINGS-protocol daemon.
Mar 23 14:35:18 purplehippo xsettingsd[469119]: xsettingsd: Loaded 14 settings from /home/avery/.config/xsettingsd/xsettingsd.conf
Mar 23 14:35:18 purplehippo xsettingsd[469119]: xsettingsd: Created window 0xc00001 on screen 0 with timestamp 5373106
Mar 23 14:35:18 purplehippo xsettingsd[469119]: xsettingsd: Selection _XSETTINGS_S0 is owned by 0x0
Mar 23 14:35:18 purplehippo xsettingsd[469119]: xsettingsd: Took ownership of selection _XSETTINGS_S0
Mar 23 14:53:07 purplehippo xsettingsd[469119]: xsettingsd: Reloading configuration
Mar 23 14:53:07 purplehippo xsettingsd[469119]: xsettingsd: Loaded 14 settings from /home/avery/.config/xsettingsd/xsettingsd.conf

If it’s having an issue, try restarting it, or you can examine the configuration file it creates automatically in your $XDG_CONFIG_HOME dir and make sure it all looks reasonable:

❯ bat $XDG_CONFIG_HOME/xsettingsd/xsettingsd.conf
      β”‚ File: /home/avery/.config/xsettingsd/xsettingsd.conf
  1   β”‚ Gdk/UnscaledDPI 98304
  2   β”‚ Gdk/WindowScalingFactor 1
  3   β”‚ Gtk/EnableAnimations 1
  4   β”‚ Gtk/DecorationLayout "icon:minimize,maximize,close"
  5   β”‚ Net/ThemeName "adw-gtk3-dark"
  6   β”‚ Gtk/PrimaryButtonWarpsSlider 0
  7   β”‚ Gtk/ToolbarStyle 3
  8   β”‚ Gtk/MenuImages 1
  9   β”‚ Gtk/ButtonImages 1
 10   β”‚ Gtk/CursorThemeSize 64
 11   β”‚ Gtk/CursorThemeName "Posy_Cursor_125_175"
 12   β”‚ Net/SoundThemeName "ocean"
 13   β”‚ Net/IconThemeName "Papirus"
 14   β”‚ Gtk/FontName "Noto Sans,  10"
 15   β”‚ 

if THAT all looks fine, there was one last thing on wiki.archlinux.org that looked like it might work (especially with Window Manager-type setups like minimal tiling desktop such as sway and dwm – although, if you’re into minimal WMs, not sure if you’d like flatpaks – in any event, nobody’s a monolith…). You could put something like this in $HOME/bin and call it refresh-flatpak-themes (etc.) then run it when you see one getting dodgy:

#!/usr/bin/env bash
for FLATPAK_APP in "$HOME/.var/app/*"; do
  [ -d "$HOME/.var/app/$FLATPAK_APP/config/gtk-3.0" ] ||
  /usr/bin/ln -s "$HOME/.config/gtk-3.0" "$HOME/.var/app/$FLATPAK_APP/config/"

Which basically just loops through each flatpak app you have in your user’s .var folder, checks to see if it has a gtk-3.0 definition folder, and if not, makes a symbolic link to where one usually lives in your $XDG_CONFIG_HOME. You could try making the symlink to one manually to see if it works before setting up a script, but if it works out, hey, go for it.

Another option I saw which looks pretty elegant is outlined here on a xerolinux-specific BB, but should apply to pretty much any distro using the xdg user-dirs spec:

❯ export GTK3_THEME=(name of preferred theme);
  flatpak --user override --filesystem=$HOME/.local/share/themes;
  flatpak --user override --filesystem=$XDG_CONFIG_HOME/gtk-3.0:ro;
  for FLATPAK_APP in $HOME/.var/app/*; do flatpak --user override \

Quick pointers for troubleshooting:

  1. The user-dirs.dirs file usually lives at $HOME/.config/user-dirs.dirs
  2. Any $VARIABLE can be echoed to the screen using echo $VARNAME (make sure they’re properly set)