Ninkasi Bug Fixes

I spent a bit of time hacking on Ninkasi code this weekend!

I got rid of one terrible idea I implemented, replaced it with something much better, fixed one crash bug, and two potential coroutine-related memory leaks!

Removing One Ugly Anti-feature

First order of business was to change the object-function-call stupidity I put in there forever ago. It's an anti-feature I never should have considered. Here's how it went:

An object can have values inside of it assigned to functions, like so:

function testFunc(testValue)
    print("Value passed in: ", testValue, "\n");

var ob = object();
ob["foo"] = testFunc;

Previously, these functions can be called like so:;

When called in this manner, the object itself is inserted as the first argument to the function. This is roughly equivalent to the following code:

    var _temp =;

And the output of this is, predictably:

Value passed in: <object:0>

The problem with this is that there's no way to distinguish between when you want the object to be inserted as the first argument to a function and when you do not.

My reasoning at the time I developed this "feature" was sketchy at best, so I finally got around to cutting it out. Now there is no special case considered for calling a function directly from an object's field using '.'.

Now this code...

function testFunc(testValue)
    print("Value passed in: ", testValue, "\n");

var ob = object();
ob["foo"] = testFunc;;

... is a runtime error, as it should be:

test.nks:14: Incorrect argument count for function call. Expected: 1 Found: 0

To replace this feature, because still like some aspects of it, I have stolen and repurposed C's indirection operator. The following code acts the way acted before:


Which is equivalent to:;

This gets us back to the original result:

Value passed in: <object:0>

But now you get to decide if you want to call a function on an object without the object itself being added in as an argument, by using the -> or the ..

One Crash Bug

Callable Objects

First, an explanation of the feature this affected:

Callable objects in Ninkasi are a little oddity I thought it would be useful to throw in. It's sort of a halfway point to actual closures.

Objects may be called as functions through the use of specially-named fields.

Take the following code:

var ob = newobject;
ob._exec = function(foo) {
    print("foo: ", foo, "\n");
ob._data = "blah blah blah";

The output of this code is:

foo: blah blah blah

It's that simple. Set an object's _exec field to a function, set an object's _data field to a value, and you may call the object itself as though it were a function.

This code...


... is equivalent to this code...


You can even add extra arguments after the _data one, which is inserted first:

var ob = object();
ob._exec = function(foo, bar) {
    print("foo: ", foo, ", ", bar, "\n");
ob._data = "blah blah blah";


foo: blah blah blah, whatever

The Bug

This feature does, unfortunately, involve some weird shifting-around of stack data.

This part wasn't thought out super-well, so bare with me here.

The setup for the function-call operator involves pushing the function ID onto the stack, followed by all the function arguments, followed by an integer indicating the number of arguments there are.

Here's what the stack will look like for a normal 3-parameter function call, at the time the call opcode executes (with stack indices relative to the top of the stack):

index | value       | type
-6    | ...         |
-5    | function-ID | function
-4    | arg0        | any type
-3    | arg1        | any type
-2    | arg2        | any type
-1    | 3           | integer

We have the function ID, followed by the arguments, followed by an argument count.

In order for the _data field to always be first, we need the stack to look like this:

index | value       | type
-7    | ...         |
-6    | function-ID | function
-5    | _data       | any type
-4    | arg0        | any type
-3    | arg1        | any type
-2    | arg2        | any type
-1    | 4           | integer

That just involves inserting the _data into the stack, shifting everything forward by 1, and adding 1 to the argument count entry.

I know this is probably tickling everyone's inefficient-VM sense, but this feature was kind of an afterthought for my personal toy language, so be nice. :)

Anyway, our bug today was in the code to shift the stack data around. The argument count coming into the function call opcode could be larger than the stack, and it would happily underrun the bottom of the stack, leading to your usual C buffer overrun/underrun type of error. That code is also not the prettiest code in the world, leading me to think I was either tired or in a rush when I wrote it. Let's have a look...

// Make room for the _data contents on the stack.

// Shift everything up the stack.
for(i = argumentCount; i >= 1; i--) {
    vm->currentExecutionContext->stack.size - argumentCount - 2 + i + 1] =
      vm->currentExecutionContext->stack.size - argumentCount - 2 + i];


How to Setup This Bug

So here's an interesting thing: You can't.

Well, let me be more clear, you can't if you're using the scripting language directly. This is purely a vulnerability exposed through the binary state snapshots.

I haven't actually reverse-engineered AFL's crash output for this one, to be entirely honest, but my thinking is that the bad executable just pushes a bad argument count onto the stack before doing a normal callable-object call.

The Fix

First, I know what you're going to say. How dare I not have a system in place to protect against these kinds of overrun/underruns, when it's so easy to check for!? That usually comes after "why the hell are you using C89 for this project when it's absolutely prone to these kinds of errors!?" followed by a recommendation that I switch to Rust.

The thing is, though, I do have a way of safely interacting with the stack. I'm just not using it here because... uh... reasons?

The stoage space allocated to the stack only doubles in size when it reallocates, so, given that the stack capacity is always a power of two, it's pretty easy to just mask off the lowest some-number-of bits for stack indices going into the array. I even store that stack mask, and use it like I actually know what I'm doing in most spots!

The fix (along with some much-needed cleanup) look like this:

struct NKVMStack *stack = &(vm->currentExecutionContext->stack);
struct NKValue *stackValues;
nkuint32_t stackSize;
nkuint32_t stackMask;
nkuint32_t stackSizeMinusArguments;

// Make room for the _data contents on the stack.

stackValues = stack->values;
stackSize = stack->size;
stackMask = stack->indexMask;
stackSizeMinusArguments = stackSize - argumentCount - 2;

// Shift everything up the stack.
for(i = argumentCount; i >= 1; i--) {
    stackValues[(stackSizeMinusArguments + i + 1) & stackMask] =
      stackValues[(stackSizeMinusArguments + i) & stackMask];

This also throws in a few extra variables to cut down on those giant globs of indirection and indexing inside the loop, potentially offering a speedup on compilers that can't optimize out those redundant operations (not that it's a particularly fast piece of code to begin with).

Memory Leaks

I'm not going to get into too much detail with this one, because it's pretty simple. A failure during coroutine creation would make the program allocate the storage space needed for the coroutine's execution context (including stack, program counter, etc) and then fail to set the pointer to store it, causing it to leak.

The fix is just to check for those failures and clean up the mess when it does.

Memory still Tracked

So with this leak still have a backup system in place for these. Allocations inside the VM are actually tracked and easily cleaned up during deallocation of the VM object itself. It's not a C memory leak, it's just a leak of VM address space, more or less.

When the VM's address space is exhausted, it will bail out with its own internal allocation error and can still be cleaned up by the hosting application. So as far as the stability of the hosting application is concerned, and in relation to potentially hostile script code, this isn't a "real" threat. It does suck for long-running scripts, though.

Final Thoughts

AFL is great. Valgrind is awesome, too. All these issues found today were through those two tools. Thinking you can write secure code without them or some equivalent to them is hubris in its purest form.

The issues with coroutine setup here are definitely showing my lack of testing after writing the coroutine code. The callable object issues are something I wasn't expecting, because that code's been in there forever.

Ninkasi may never be perfect and vulnerability-proof, but at least it got a little bit closer today, and I'm pretty happy with that.

Posted: 2021-09-05

Grafx2 Panning

I've been spending the last year working (off-and-on) on some changes to Grafx2!

Grafx2, for anyone not aware of it, is an old piece of pixel art software from the 90s, for doing 256 color art. It closely mimics Deluxe Paint, for those familiar with it.

There are a few weird aspects of the UI (attributable to the fact that it's from the 90s), but the biggest issue I have with it is the limited ability to pan around the image. As of this writing, Grafx2 doesn't let you move the visible area in a way that would let you center the upper-left corner of the image. You can't pan further to the left than the point where the left edge touches the left side of the screen (or window). Similarly for the top edge and vertical panning.

Grafx2 without panning

Thankfully, Grafx2 is also open-source! So I figured "oh! Why don't I just go in and fix that?"

The result was a lot more work than I bargained for. But it's finally starting to show results. You can now pan the image around a lot more freely in Grafx2!

Grafx2 with panning

Grafx2 with panning

I have a branch that, while the code is SUPER MESSY and spills out a ton of warnings, it's still functional. Almost everything works (except one of the many pixel scaling modes, as of this writing).

My current "panning" branch is available right here. Feel free to give it a whirl! Please, please, please let me know if you find any bugs specific to the panning branch!

The messy details

This change touches a LOT of code.

Remaining work

The code is extremely messy at the moment, and needs a ton of cleanup, but everything is currently working. At least, on PC.

It still needs testing on all the other platforms Grafx2 runs on. This is largely due to switching around some integer data types that might end up with different sizes or signedness on other platforms with different data models.

Posted: 2021-08-22

Deluxe Paint Gradient Fills

So I watched Mark Ferrari's entire GDC talk "8 Bit & '8 Bitish' Graphics-Outside the Box". It's a fantastic talk by one of the greatest pixel artists alive today. He talks about a lot of his work, and spends a section of the video going into some details about his techniques.

I tried to follow along as best I could in his tutorial sections using Grafx2. Unfortunately, there are a few things that Deluxe Paint and/or Pro Motion can do that Grafx2 still can't. One of those things I'd really like to use is Deluxe Paint's gradient fill modes.

So I've been trying to dissect the way Deluxe Paint does its shape-conforming gradient fills, and seeing if I can implement them myself in Grafx2. (Grafx2 also a has an existing feature request that mentions this functionality: Add more flexible gradients)

Here's what I'm up against:

Menu of different Deluxe Paint fill modes


"Contour" fill

Contour fill

The key here to replicating some of DP's odd behavior is that it's always using the most distant edge. This is why DP's fills look a little weird when the fill wraps around curves (where it has multiple edges to choose from along some lines coming from the center.

From our top-right example, we can see how this breaks down. The gradient is being done based on the most distant edge, so it's affected by the hook shape.

Contour fill example 2

And the bottom-right example. We can clearly see where it's preferring far edges to interpolate to.

Contour fill example 3

"Highlight" fill

I believe this is identical to "Countour", but with a different mapping to the actual gradient. t may be affected by a curve, similar to a gamma curve. So I'm not going to cover this one here.

"Ridges" fill

Ridges fill

"Linear" fill

This is a pretty basic linear gradient fill. Grafx2 already supports this, so it's only here for completeness.

"Circular" fill

This is a basic radial gradient. Grafx2 already does this, too, so we won't get into it.

"Shape" fill

Shaped fill

This one is like the "contour" fill, except that it uses a line instead of a single centerpoint. We'll do the same math as before, but with an extra projection. Rather than a single mid-point, as used in the "contour" mode, we need to use an entire line.

Shaped fill description example

It'll work exactly the same as the contour gradient, except that we need to find the closest point along a given center line, and use that as the A vector in the equation.


So far I've only worked on figuring out the "contour" fill mode. My results are looking pretty promising, though! Here's what my prototype spits out with the same image input (slightly different points selected as centers):

My own contour gradient fill implementation results

It doesn't match the DP results exactly, but that's more a matter of correctly mapping the gradient to color values than anything to do with the technique.

Next update will probably focus on the "shaped" mode and optimizations.

Posted: 2021-08-09

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 [ 30 ] 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68

Previous | Next