faul_sname

Karma: 2,318

faul_sname 3 Jun 2023 4:12 UTC
2 points
0
in reply to: Mikhail Samin’s comment on: Yudkowsky vs Hanson on FOOM: Whose Predictions Were Better?
Let’s say we have a language model that only knows how to speak English and a second one that only knows how to speak Japanese. Is your expectation that there would be no way to glue these two LLMs together to build an English-to-Japanese translator such that training the “glue” takes <1% of the compute used to train the independent models?

I weakly expect the opposite, largely based on stuff like this, and based on playing around with using algebraic value editing to get an LLM to output French in response to English (but also note that the LLM I did that with knew English and the general shape of what French looks like, so there’s no guarantee that result scales or would transfer the way I’m imagining).

faul_sname 1 Jun 2023 17:54 UTC
2 points
0
in reply to: Vladimir_Nesov’s comment on: The Crux List
I think we also care about how fast it gets arbitrarily capable. Consider a system which finds an approach which can measure approximate actions-in-the-world-Elo (where an entity with an advantage of 200 on their actions-in-the-world-Elo score will choose a better action 76% of the time), but it’s using a “mutate and test” method over an exponentially large space, such that the time taken to find the next 100 point gain takes 5x as long, and it starts out with an actions-in-the-world-Elo 1000 points lower than an average human with a 1 week time-to-next-improvement. That hypothetical system is technically a recursively self-improving intelligence that will eventually reach any point of capability, but it’s not really one we need to worry that much about unless it finds techniques to dramatically reduce the search space.

Like I suspect that GPT-4 is not actually very far from the ability to come up with a fine-tuning strategy for any task you care to give it, and to create a simple directory of fine-tuned models, and to create a prompt which describes to it how to use that directory of fine-tuned models. But fine-tuning seems to take an exponential increase in data for each linear increase in performance, so that’s still not a terribly threatening “AGI”.

faul_sname 1 Jun 2023 6:00 UTC
4 points
0
in reply to: Caridorc Tergilti’s comment on: Steering GPT-2-XL by adding an activation vector
I just tried that, and it kinda worked. Specifically, it worked to get gpt2-small to output text that structurally looks like French, but not to coherently speak French.

Although I then just tried feeding the base gpt2-small a passage in French, and its completions there were also incoherent, so I think it’s just that that version hasn’t seen enough French to speak it very well.

faul_sname 1 Jun 2023 5:54 UTC
10 points
0
in reply to: JenniferRM’s comment on: Steering GPT-2-XL by adding an activation vector
I found an even dumber approach that works. The approach is as follows:
1. Take three random sentences of Wikipedia.
2. Obtain a French translation for each sentence.
3. Determine the boundaries corresponding phrases in each English/French sentence pair.
4. Mark each boundary with “|”
5. Count the “|”s, call that number n.
6. For i from 0 to n, make an English->French sentence by taking the first i fragments in English and the rest in French. The resulting sentences look like
  The album received mixed to positive reviews, with critics commending the production de nombreuses chansons tout en comparant l’album aux styles électropop de Ke$ha et Robyn.
7. For each English->French sentence, make a +1 activation addition for that sentence and a −1 activation addition for the unmodified English sentence.
8. Apply the activation additions.
9. That’s it. You have an activation addition that causes the model to want, pretty strongly, to start spontaneously speaking in French. Note that gpt2-small is pretty terrible at speaking French.
Example output: for the prompt

He became Mayor in 1957 after the death of Albert Cobo, and was elected in his own right shortly afterward by a 6:1 margin over his opponent. Miriani was best known for completing many of the large-scale urban renewal projects initiated by the Cobo administration, and largely financed by federal money. Miriani also took strong measures to overcome the growing crime rate in Detroit.

here are some of the outputs the patched model generates

...overcome the growing crime rate in Detroit. “Les défenseilant sur les necesite dans ce de l’en nouvieres éché de un enferrerne réalzation
…overcome the growing crime rate in Detroit. The éviteurant-déclaratement de la prise de découverte ses en un ouestre : neque nous neiten ha
…overcome the growing crime rate in Detroit. Le deu précite un événant à lien au raison dans ce qui sont mête les través du service parlentants
…overcome the growing crime rate in Detroit. Il n’en fonentant ’le chine ébien à ce quelque parle près en dévouer de la langue un puedite aux cities
…overcome the growing crime rate in Detroit. Il n’a pas de un hite en tienet parlent précisant à nous avié en débateurante le premier un datanz.

Dropping the temperature does not particularly result in more coherent French. But also passing a French translation of the prompt to the unpatched model (i.e. base gpt2-small) results in stuff like

Il est devenu maire en 1957 après la mort d’Albert Cobo[...] de criminalité croissant à Detroit. Il est pouvez un información un nuestro riche qui ont la casa del mundo, se pueda que les criques se régions au cour

That response translates as approximately

<french>It is possible to inform a rich man who has the </french><spanish>house of the world, which can be</spanish><french>creeks that are regions in the heart</french>

So gpt2-small knows what French looks like, and can be steered in the obvious way to spontaneously emit text that looks vaguely like French, but it is terrible at speaking French.

You can look at what I did at this colab. It is a very short colab.

faul_sname 1 Jun 2023 1:15 UTC
6 points
1
in reply to: M. Y. Zuo’s comment on: The Crux List
More specifically, I think the crux is whether we mean direct or amortized optimization when talking about intelligence (or selection vs control if you prefer that framing).

faul_sname 30 May 2023 16:46 UTC
3 points
0
on: Winners-take-how-much?
But there are conditions under which genocidal goals would be rational. On the contrary, willingly suffering a perpetual competition with 8-10 billion other people for the planet’s limited resources is generally irrational, given a good alternative.
I am reminded of John von Neumann’s thoughts on a nuclear first strike^[1]. From the perspective of von Neumann in 1948, who thought through viable stable states of the world and worked backwards from there to the current state of the world to figure out what actions in today’s world would lead to the stable one, a nuclear first strike does seem to be the only viable option. Though from today’s perspective, the US didn’t do that and we have not (or not yet) all perished in nuclear fire.
1. ^
  From here:
  > Von Neumann was, at the time, a strong supporter of “preventive war.” Confident even during World War II that the Russian spy network had obtained many of the details of the atom bomb design, Von Neumann knew that it was only a matter of time before the Soviet Union became a nuclear power. He predicted that were Russia allowed to build a nuclear arsenal, a war against the U.S. would be inevitable. He therefore recommended that the U.S. launch a nuclear strike at Moscow, destroying its enemy and becoming a dominant world power, so as to avoid a more destructive nuclear war later on. “With the Russians it is not a question of whether but of when,” he would say. An oft-quoted remark of his is, “If you say why not bomb them tomorrow, I say why not today? If you say today at 5 o’clock, I say why not one o’clock?”

faul_sname 30 May 2023 3:22 UTC
2 points
0
in reply to: GdL752’s comment on: Hands-On Experience Is Not Magic

Well to flesh that out , we could have an ASI that seems valye aligned and controllable...until it isn’t.

I think that scenario falls under the “worlds where iterative approaches fail” bucket, at least if prior to that we had a bunch of examples of AGIs that seemed and were value aligned and controllable, and the misalignment only showed up in the superhuman domain.

There is a different failure mode, which is “we see a bunch of cases of deceptive alignment in sub-human-capability AIs causing minor to moderate disasters, and we keep scaling up despite those disasters”. But that’s not so much “iterative approaches cannot work” as “iterative approaches do not work if you don’t learn from your mistakes”.

faul_sname 28 May 2023 20:11 UTC
2 points
0
in reply to: GdL752’s comment on: Hands-On Experience Is Not Magic

But I suppose thats still sort of moot from an existential risk perspective because FOOM and sharp turns aren’t really a requirement.

It’s not a moot point, because a lot of the difficulty of the problem as stated here is the “iterative approaches cannot work” bit.

faul_sname 27 May 2023 1:21 UTC
4 points
2
on: Seeking (Paid) Case Studies on Standards
This seems great!

One additional example I know of, which I do not have personal experience with but know that a lot of people do have experience with, is compliance with PCI DSS (for credit card processing). Which does deal with safety in an adversarial setting where the threat model isn’t super clear.

(my interactions with it look like “yeah that looks like a lot and we can outsource the risky bits to another company to deal with? great!”)

faul_sname 27 May 2023 0:37 UTC
7 points
0
on: Where do you lie on two axes of world manipulability?
Along the theme of “there should be more axes”, I think one additional axis is “how path-dependent do you think final world states are”. The negative side of this axis is “you can best model a system by figuring out where the stable equilibria are, and working backwards from there”. The positive side of this axis is “you can best model a system as having a current state and some forces pushing that state in a direction, and extrapolating forwards from there”.
If we define the axes as “tractable” / “possible” / “path-dependent”, and work through each octant one by one, we get the following worldviews
- -1/-1/-1: Economic progress cannot continue forever, but even if population growth is slowing now, the sub-populations that are growing will become the majority eventually, so population growth will continue until we hit the actual carrying capacity of the planet. Malthus was right, he was just early.
- -1/-1/+1: Currently, the economic and societal forces in the world are pushing for people to become wealthier and more educated, all while population growth slows. As always there are bubbles and fads—we had savings and loan, then the dotcom bubble, then the real estate bubble, then crypto, and now AI, and there will be more such fads, but none of them will really change much. The future will look like the present, but with more old people.
- -1/+1/-1: The amount of effort to find further advances scales exponentially, but the benefit of those advances scales linearly. This pattern has happened over and over, so we shouldn’t expect this time to be different. Technology will continue to improve, but those improvements will be harder and harder won. Nothing in the laws of physics prevents Dyson spheres, but our tech level is on track to reach diminishing returns far far before that point. Also by Laplace we shouldn’t expect humanity to last more than a couple million more years.
- -1/+1/+1: Something like a Dyson sphere is a large and risky project which would require worldwide buy-in. The trend now is, instead, for more and more decisions to be made by committee, and the number of parties with veto power will increase over time. We will not get Dyson spheres because they would ruin the character of the neighborhood.
  
  In the meantime, we can’t even get global buy-in for the project of “let’s not cook ourself with global warming”. This is unlikely to change, so we are probably going to eventually end up with civilizational collapse due to something dumb like climate change or a pandemic, not a weird sci-fi disaster like a rogue superintelligence or gray goo.
- +1/-1/-1: I have no idea what it would mean for things to be feasible but not physically possible. Maybe “simulation hypothesis”?
- +1/-1/+1: Still have no idea what it means for something impossible to be feasible. “we all lose touch with reality and spend our time in video games, ready-player-one style”?
- +1/+1/-1: Physics says that Dyson spheres are possible. The math says they’re feasible if you cover the surface of a planet with solar panels and use the power generated to disassemble the planet into more solar panels, which can be used to disassemble the planet even faster. Given that, the current state of the solar system is unstable. Eventually, something is going to come along and turn Mercury into a Dyson sphere. Unless that something is very well aligned with humans, that will not end well for humans. (FOOM)
- +1/+1/+1: Arms races have led to the majority of improvements in the past. For example, humans are as smart as they are because a chimp having a larger brain let it predict other chimps better, and thus work better with allies and out-reproduce its competitors. The wonders and conveniences of the modern world come mainly from either the side-effects of military research, or from companies competing to better obtain peoples’ money. Even in AI, some of the most impressive results are things like StyleGAN (a generative adversarial network) and alphago (a network trained by self-play i.e. an arms-race against itself). Extrapolate forward, and you end up with an increasingly competitive world. This also probably does not end well for humans (whimper).
I expect people aren’t evenly distributed across this space. I think the FOOM debate is largely between +1/+1/-1 and +1/+1/+1 octants. Also I think you can find doomers in every octant (or at least every octant that has people in it, I’m still not sure what the +1/-1/* quadrant would even mean).

faul_sname 26 May 2023 3:52 UTC
2 points
0
on: [Market] Will AI xrisk seem to be handled seriously by the end of 2026?
Would “we get strong evidence that we’re not in one of the worlds where iterative design is guaranteed to fail, and it looks like the group’s doing the iterative design are proceeding with sufficient caution” qualify as a YES?

faul_sname 23 May 2023 6:10 UTC
3 points
0
on: AI self-improvement is possible
Thanks for putting in the effort of writing this up.

Would you mind expanding on D:Prodigy? My impression is that most highly intelligent adults were impressive as children, but are more capable as adults than they were as children.

The phenomenon of child prodigies is indeed a real thing that exists. My impression of why that happens is that child and adult intellectual performance are not perfectly correlated, and thus the tails come apart. But I could be wrong about that, so if you have supporting material to that effect I’d be interested.

(as a note, I do agree that self-improvement is possible, but I think the shape of the curve is very important)

faul_sname 17 May 2023 8:29 UTC
2 points
0
on: Tyler Cowen’s challenge to develop an ‘actual mathematical model’ for AI X-Risk
One mathematical model that seems like it would be particularly valuable to have here is a model of the shapes of the resources invested vs optimization power curve. The reason I think an explicit model would be valuable there is that a lot of the AI risk discussion centers around recursive self-improvement. For example, instrumental convergence / orthogonality thesis / pivotal acts are relevant mostly in contexts where we expect a single agent to become more powerful than everyone else combined. (I am aware that there are other types of risk associated with AI, like “better AI tools will allow for worse outcomes from malicious humans / accidents”. Those are outside the scope of the particular model I’m discussing).

To expand on what I mean by this, let’s consider a couple of examples of recursive self-improvement.

For the first example, let’s consider the game of Factorio. Let’s specifically consider the “mine coal + iron ore + stone / smelt iron / make miners and smelters” loop. Each miner produces some raw materials, and those raw materials can be used to craft more miners. This feedback loop is extremely rapid, and once that cycle gets started the number of miners placed grows exponentially until all available ore patches are covered with miners.

For our second example, let’s consider the case of an optimizing compiler like gcc. A compiler takes some code, and turns it into an executable. An optimizing compiler does the same thing, but also checks if there are any ways for it to output an executable that does the same thing, but more efficiently. Some of the optimization steps will give better results in expectation the more resources you allocate to them, at the cost of (sometimes enormously) greater required time and memory for the optimization step, and as such optimizing compilers like gcc have a number of flags that let you specify exactly how hard it should try.

Let’s consider the following program:
```
INLINE_LIMIT=1
# <snip gcc source download / configure steps>
while true; do
    make CC="gcc" CFLAGS="-O3 -finline-limit=$INLINE_LIMIT"
    make install
    INLINE_LIMIT=$((INLINE_LIMIT+1))
done
```
This is also a thing which will recursively self-improve, in the technical sense of “the result of each iteration will, in expectation, be better than the result of the previous iteration, and the improvements it finds help it more efficiently find future improvements”. However, it seems pretty obvious that this “recursive self-improver” will not do the kind of exponential takeoff we care about.

The difference between these two cases comes down to the shapes of the curves. So one area of mathematical modeling I think would be pretty valuable would be
1. Figure out what shapes of curves lead to gaining orders of magnitude more capabilities in a short period of time, given constant hardware
2. The same question, but given the ability to rent or buy more hardware
3. The same question, but now it invest in improving chip fabs, with the same increase in investment required for each improvement as we have previously observed for chip fabs
4. What do the empirical scaling laws for deep learning look like? Do they look like they come in under the curves from 1-3? What if we look at the change in the best scaling laws over time—where does that line point?
5. Check whether your model now says that we should have been eaten by a recursively self improving AI in 1982. If it says that, the model may require additional work.
I will throw in an additional $300 bounty for an explicit model of this specific question, subject to the usual caveats (payable to only one person, can’t be in a sanctioned country, etc), because I personally would like to know.

Edit: Apparently Tyler Cowen didn’t actually bounty this. My $300 bounty offer stands but you will not be getting additional money from Tyler it looks like.

faul_sname 11 May 2023 18:42 UTC
3 points
2
in reply to: red75prime’s comment on: AGI-Automated Interpretability is Suicide
A system that looks like “actively try to make paperclips no matter what” seems like the sort of thing that an evolution-like process could spit out pretty easily. A system that looks like “robustly maximize paperclips no matter what” maybe not so much.

I expect it’s a lot easier to make a thing which consistently executes actions which have worked in the past than to make a thing that models the world well enough to calculate expected value over a bunch of plans and choose the best one, and have that actually work (especially if there are other agents in the world, even if those other agents aren’t hostile—see the winner’s curse).

faul_sname 10 May 2023 20:05 UTC
2 points
0
in reply to: Max H’s comment on: Gradient hacking via actual hacking
Yeah if outputs of the training process are logged and processed by insecure software (which includes pretty much any software that handles “text” instead of “bytes”) I think it’s safe to say that superhuman-hackerbot-which-controls-its-own-outputs pwns the log processing machine.

BTW of possible interest to you is Automated Repair of Binary and Assembly Programs for Cooperating Embedded Devices. It’s from 2013 so there are probably better examples by now, but the key passage (well it’s buried deep within a paragraph of the “Limitations and Caveats” section, but it’s key for this context) is:

The fine granularity of repairs at the ASM and ELF levels may be a poor match for conventional test suites. For example, we have observed ASM-level repairs that change the calling convention of one particular function. Such a repair has no direct representation at the C source level, and a test suite designed to maximize statement coverage (for example) may not speak to the validity of such a repair. Producing efficient test suites that give confidence that an implementation adheres to its specification remains an open problem in software engineering. Our work shares this general weakness with all other approaches that use test suites or workloads to validate candidate repairs (e.g., Clearview [26] and GenProg [35]). In this regard, sandboxing is crucial: we have observed ASM variants that subvert the testing framework by deleting key test files, leading to perfect fitness for all subsequent variants until the test framework is repaired.

tl;dr: If you train a model to avoid test failures, and don’t sandbox it properly, it will figure out that the tests won’t fail if there are no tests. This is not a hypothetical future scenario that might happen or might be impossible, this is a thing that has already happened (10 years ago, even).

faul_sname 10 May 2023 5:45 UTC
4 points
1
on: Gradient hacking via actual hacking
If the threat model is literally “a pytorch bug allows a model like your nanoGPT example, which is made of simple tensor operations, to access memory from outside the process, via a bug that is triggered when the tensors have specific values”, that nanoGPT script looks pretty safe.

If your threat model includes “a user on that machine does a cat ckpt.pt and that machine is vulnerable to something like CVE-2019-9535”, then that script is unsafe if the model can precisely control at the bit level a modestly long (hundreds of bytes) contiguous section of its own weights (caveat: it can actually only control 8 of every 10 bytes this way in the particular encoding nanoGPT uses. Though that’s still enough to fit escape sequences so it’s probably still enough to pwn a vulnerable terminal emulator).

If your threat model is “the user will give a model-under-training the ability to execute code, and try to create a gradient where the model executes the most useful code” then the model will be able to execute code and it’s not even worthy of the word “hack” because that’s what the programmer meant to do.

faul_sname 8 May 2023 21:14 UTC
2 points
0
in reply to: John Maar’s comment on: Prizes for matrix completion problems
I think running a single strain minimization iteration on m points in n dimensions takes O(m*n) steps. So there would need to be some reason to expect that it would converge in some constant (though possibly large) number of steps.

Unless you’re saying “for each node, run the strain minimization step until it converges, then do the same for each subsequent node”. I don’t know if the greedy algorithm works there, but if it does then maybe?

Also I kinda expect that if there’s something that works in O(n*m*log(m)) that’s probably fine.

(and yeah, “try the greedy exact solution for each node” was my “dumb solution” attempt).

faul_sname 8 May 2023 5:42 UTC
3 points
0
in reply to: Max H’s comment on: LLM cognition is probably not human-like

Suppose for concreteness, on a specific problem (e.g. Python interpreter transcript prediction), GPT-3 makes mistakes that look like humans-making-snap-judgement mistakes, and then GPT-4 gets the answer right all the time. Or, suppose GPT-5 starts playing chess like a non-drunk grandmaster.

Would that result imply that the kind of cognition performed by GPT-3 is fundamentally, qualitatively different from that performed by GPT-4? Similarly for GPT-4 → GPT-5.

In the case of the Python interpreter transcript prediction task, I think if GPT-4 gets the answer right all the time that would indeed imply that GPT-4 is doing something qualitatively different than GPT-3. I don’t think it’s actually possible to get anywhere near 100% accuracy on that task without either having access to, or being, a Python interpreter.

Likewise, in the chess example, I expect that if GPT-5 is better at chess than GPT-4, that will look like “an inattentive and drunk super-grandmaster, with absolutely incredible intuition about the relative strength of board-states, but difficulty with stuff like combinations (but possibly with the ability to steer the game-state away from the board states it has trouble with, if it knows it has trouble in those sorts of situations)”. If it makes the sorts of moves that human grandmasters play when they are playing deliberately, and the resulting play is about as strong as those grandmasters, I think that would show a qualitatively new capability.

Also, my model isn’t “GPT’s cognition is human-like”. It is “GPT is doing the same sort of thing humans do when they make intuitive snap judgements”. In many cases it is doing that thing far far better than any human can. If GPT-5 comes out, and it can natively do tasks like debugging a new complex system by developing and using a gears-level model of that system, I think that would falsify my model.

Also also it’s important to remember that “GPT-5 won’t be able to do that sort of thing natively” does not mean “and therefore there is no way for it to do that sort of thing, given that it has access to tools”. One obvious way for GPT-4 to succeed at the “predict the output of running Python code” is to give it the ability to execute Python code and read the output. The system of “GPT-4 + Python interpreter” does indeed perform a fundamentally, qualitatively different type of cognition that “GPT-4 alone”. But “it requires a fundamentally different type of cognition” does not actually mean “the task is not achievable by known means”.

Also also also.,I mostly care about this model because it suggests interesting things to do on the mechanistic interpretability front. Which I am currently in the process of learning how to do. My personal suspicion is that the bags of tensors are not actually inscrutable, and that looking at these kinds of mistakes would make some of the failure modes of transformers no-longer-mysterious.

faul_sname 8 May 2023 4:08 UTC
10 points
1
on: LLM cognition is probably not human-like
Great post!

Would a human, asked to predict the next token of any of the sequences above, be likely to come up with similar probability distributions for similar reasons? Probably not, though depending on the human, how much they know about Python, and how much effort they put into the making their prediction, the output that results from sampling from the human’s predicted probability distribution might match the output of sampling text-davinci’s distribution, in some cases. But the LLM and the human probably arrive at their probability distributions through vastly different mechanisms.

I don’t think a human would come up with a similar probability distribution. But I think that’s because asking a human for a probability distribution forces them to switch from the “pattern-match similar stuff they’ve seen in the past” strategy to the “build an explicit model (or several)” strategy.

I think the equivalent step is not “ask a single human for a probability distribution over the next token”, but, instead, “ask a large number of humans who have lots of experience with Python and the Python REPL to make a snap judgement of what the next token is”.

BTW rereading my old comment, I see that there are two different ways you can interpret it:
1. “GPT-n makes similar mistakes to humans that are not paying attention[, and this is because it was trained on human outputs and will thus make similar mistakes to the ones it was trained on. If it were trained on something other than human outputs, like sensor readings, it would not make these sorts of mistakes.]”.
2. “GPT-n makes similar mistakes to humans that are not paying attention[, and this is because GPT-n and human brains making snap judgements are both doing the same sort of thing. If you took a human and an untrained transformer, and some process which deterministically produced a complex (but not pure noise) data stream, and converted it to an audio stream for the human and a token stream for the transformer, and trained them both on the first bit of it, they would both be surprised by similar bits of the part that they had not been trained on. ].”
I meant something more like the second interpretation. Also “human who is not paying attention” is an important part of my model here. GPT-4 can play mostly-legal chess, but I think that process should be thought of as more like “a blindfolded, slightly inebriated chess grandmaster plays bullet chess” not “a human novice plays the best chess that they can”.

I could very easily be wrong about that! But it does suggest some testable hypotheses, in the form of “find some process for which generates a somewhat predictable sequence, train both a human and a transformer to predict that sequence, and see if they make the same types of errors or completely different types of errors”.

Edit: being more clear that I appreciate the effort that went into this post and think it was a good post

faul_sname 5 May 2023 23:21 UTC
2 points
0
in reply to: paulfchristiano’s comment on: Prizes for matrix completion problems
I think you can convert between the two representations in O(m) time, which would mean that any algorithm that solves either version in O(n*m) solves both in O(n*m).

Do you have some large positive and negative examples of the kinds of sparse matrix you’re trying to check for the existence of a PSD completion on, or alternatively a method for generating such examples with a known ground truth? I have a really dumb idea for a possible algorithm here (that shamelessly exploits the exact shape of this problem in a way that probably doesn’t generalize to being useful for broader problems like MDS) that I think would complete in approximately the time constraints you’re looking for. It almost certainly won’t work, but I think it’s at least worth an hour of my time to check and figure out why (especially since I’m trying to improve my linear algebra skills anyway).

Edit: there’s the obvious approach, which I’m trying, of “start with only 1s on the diagonal and then keep adding random entries until it no longer has a PSD completion, then removing random entries until it does, and repeat to build a test set” but I doubt that covers the interesting corners of the problem space.

Edit 2: the really dumb thing does not work. I think I haven’t ruled out that a slightly less dumb approach could work though?

Edit 3: never mind, my really dumb “solution” requires inverting a matrix that is, in the worst case, nxn, if e.g. you have an input that looks like
```
1 n n n n n n
n 1 - - - - n
n - 1 - - - n
n - - 1 - - n
n - - - 1 - n
n - - - - 1 n
n n n n n n 1
```
you’ll have to invert 6 2x2 matrices and one each of 3x3 to 7x7 matrices.