The grep bracket trick and `exec`

A small useful trick for avoiding self-matches when using ps, grep, or pgrep to check whether a process is running.

There are actually two related fixes here:

grep '[P]attern'

and:

exec pgrep -f 'Pattern'

They solve a similar-looking problem, but they solve it in different ways.

The bracket trick changes the search pattern so the search command does not match itself. exec changes the process behavior so the temporary shell created to run the command gets replaced by the actual command.

The original issue

The original Lua code was in a configuration file in waywall_generic_config by arjuncgore for Waywall, a Wayland compositor for Minecraft speedrunning,

local is_ninb_running = function()
    local handle = io.popen("pgrep -f 'Ninjabrain.*jar'")
    local result = handle:read("*l")
    handle:close()
    return result ~= nil
end

The goal is simple:

pgrep -f 'Ninjabrain.*jar'

If a Ninjabrain jar process is running, return a PID. If it is not running, return nothing.

The issue is that pgrep -f searches the full command line of processes.

So this pattern:

Ninjabrain.*jar

should match a real process like:

java -jar Ninjabrain-Bot-1.5.1.jar

but it can also accidentally match the command that is trying to do the checking.

With io.popen, the command is usually not run directly. It is run through a shell.

So this Lua line:

io.popen("pgrep -f 'Ninjabrain.*jar'")

is roughly like running:

/bin/sh -c "pgrep -f 'Ninjabrain.*jar'"

That means the process tree can look roughly like this:

waywall
└── sh -c "pgrep -f 'Ninjabrain.*jar'"
    └── pgrep -f "Ninjabrain.*jar"

The problem is the parent shell command:

sh -c "pgrep -f 'Ninjabrain.*jar'"

That command line contains:

Ninjabrain.*jar

So pgrep -f 'Ninjabrain.*jar' can accidentally match the shell that launched it.

The script then sees a result and returns:

true

even though Ninjabrain itself is not actually running.

That is why the debug output looked like this (in my testing):

DEBUG (Original): 'pgrep -f' result = '145155'
DEBUG (Original): is_ninb_running returning: true

and:

DEBUG (Test Case 2): 'ps aux | grep java | grep Ninjabrain' result = 'jerren 145157 ... sh -c -- ps aux | grep java | grep Ninjabrain'
DEBUG (Test Case 2): is_ninb_running returning: true

The second one makes the issue very obvious, it found the original shell calling the command, not ninjabrain itself.

Fix 1: the grep bracket trick

The bracket trick looks like this:

grep '[N]injabrain'

instead of this:

grep 'Ninjabrain'

The pattern:

[N]injabrain

still matches:

Ninjabrain

because [N] means “match the character N”.

But the grep command’s own command line contains this literal text:

grep '[N]injabrain'

It does not contain this literal text:

grep 'Ninjabrain'

So it avoids matching itself.

For the Ninjabrain case, the fixed version was:

local is_ninb_running = function()
    local handle = io.popen("ps aux | grep '[N]injabrain-Bot.*\\.jar'")
    local result = handle:read("*l")
    handle:close()
    return result ~= nil
end

The shell receives this pattern:

[N]injabrain-Bot.*\.jar

That still matches a real jar process like:

java -jar Ninjabrain-Bot-1.5.1.jar

but it does not match the grep command itself, because the command line contains the bracketed version:

grep '[N]injabrain-Bot.*\.jar'

not the plain version:

grep 'Ninjabrain-Bot.*\.jar'

So when Ninjabrain is not running, it correctly returns nothing:

DEBUG (Test Case 3): 'ps aux | grep '[N]injabrain-Bot.*\.jar'' result = 'nil'
DEBUG (Test Case 3): is_ninb_running returning: false

Fix 2: using `exec`

Another fix is to keep the original pgrep version, but add exec:

local is_ninb_running = function()
    local handle = io.popen("exec pgrep -f 'Ninjabrain.*jar'")
    local result = handle:read("*l")
    handle:close()
    return result ~= nil
end

I didn’t think of this originally, but user @me_nx in the MCSR Linux discord suggested this solution.

This also fixes the issue, but for a different reason. Without exec, the command is roughly:

/bin/sh -c "pgrep -f 'Ninjabrain.*jar'"

So you may temporarily have both of these processes:

sh -c "pgrep -f 'Ninjabrain.*jar'"
pgrep -f "Ninjabrain.*jar"

The parent sh -c ... process is the risky one, because its command line contains the search pattern. With exec, the command becomes:

/bin/sh -c "exec pgrep -f 'Ninjabrain.*jar'"

The exec builtin tells the shell:

replace this shell process with the pgrep process

So instead of keeping a separate parent shell process around, the shell becomes pgrep.

Conceptually, this:

waywall
└── sh -c "exec pgrep -f 'Ninjabrain.*jar'"

turns into this:

waywall
└── pgrep -f "Ninjabrain.*jar"

There is no longer a lingering:

sh -c "pgrep -f 'Ninjabrain.*jar'"

process for pgrep -f to accidentally match. That is why this version works:

local handle = io.popen("exec pgrep -f 'Ninjabrain.*jar'")

It avoids the original problem without needing the bracket trick.

Why exec over the grep bracket trick?

The original problem depends on shell behavior: io.popen runs the command through /bin/sh. On some systems, /bin/sh may be bash. On Ubuntu, /bin/sh is usually dash. Different shells make different choices about whether they keep the shell process around or replace it with the final command in simple cases, and dash specifically chose not to replace the final command.

Some very relevant posts for more information on the differences can be read at Simon McVittie’s test case or Aviro’s StackExchange question, which detail the difference in whether they fork or replace simple commands when running commands in the respective shells.

That is why this can behave differently depending on the distro or shell:

io.popen("pgrep -f 'Ninjabrain.*jar'")

On one system, the shell may disappear quickly or optimize itself away, and on another system, the shell may remain visible long enough for pgrep -f to match it.

The exec version avoids relying on that shell-specific behavior:

io.popen("exec pgrep -f 'Ninjabrain.*jar'")

Instead of hoping the shell chooses to replace itself, you explicitly tell it to replace itself. That makes the behavior much less dependent on whether /bin/sh is bash, dash, or another POSIX-style shell.

So this is the safer mental model:

pgrep -f 'Ninjabrain.*jar'

means run pgrep through a shell and hope the shell does not get matched

while:

exec pgrep -f 'Ninjabrain.*jar'

means replace the shell with pgrep, so the shell isn't there to be matched

Bracket trick vs `exec`

The bracket trick is most useful with:

ps aux | grep ...

The exec trick is most useful when your command is being launched through a shell and you want the shell to get out of the way.

In this specific case, the exec pgrep version is probably cleaner:

local is_ninb_running = function()
    local handle = io.popen("exec pgrep -f 'Ninjabrain.*jar'")
    local result = handle:read("*l")
    handle:close()
    return result ~= nil
end

It keeps pgrep, avoids parsing ps aux, and directly solves the parent-shell self-match.

The bracket version is still a good trick to know because it works in many general ps | grep cases:

ps aux | grep '[f]irefox'
ps aux | grep '[p]ython.*server.py'
ps aux | grep '[j]ava.*my-app.*\.jar'

Another `pgrep` fix (Requires procps-ng 4.0.1+)

Important caveat: The -A flag only exists in versions of procps-ng 4.0.1 or later. That means it should be present on distros shipping procps-ng 4.x, including Ubuntu 24.04 or later, current Debian, Fedora, Arch, etc.

There is also this version:

pgrep -A -f 'Ninjabrain-Bot.*\.jar'

-A means ignore ancestor processes.

That can help because the false match is probably an ancestor of pgrep, such as:

sh -c "pgrep -f 'Ninjabrain.*jar'"

So in Lua:

local handle = io.popen("pgrep -A -f 'Ninjabrain-Bot.*\\.jar'")

That version also directly targets the ancestor-shell issue.

Conclusion

The three potential fixes I have here are the following:

-- Avoid the shell parent by replacing it with pgrep
local handle = io.popen("exec pgrep -f 'Ninjabrain.*jar'")

-- Avoid matching grep's own command line
local handle = io.popen("ps aux | grep '[N]injabrain-Bot.*\\.jar'")

-- Ask pgrep to ignore ancestor processes (Requires procps-ng 4.0.1+)
local handle = io.popen("pgrep -A -f 'Ninjabrain-Bot.*\\.jar'")

Real test example (Ubuntu 25.10)

~
❯ /bin/sh -c "pgrep -f 'discord'" | wc -l
11  # <-- +1 incorrect self detection

~
❯ /bin/sh -c "exec pgrep -f 'discord'" | wc -l
10

~
❯ /bin/sh -c "pgrep -f '[d]iscord'" | wc -l
10

~
❯ /bin/sh -c "pgrep -A -f '[d]iscord'" | wc -l
10

The grep bracket trick and exec