Scenario analysis: semi-general AIs

Are there any es­says any­where that go in depth about sce­nar­ios where AIs be­come some­what re­cur­sive/​gen­eral in that they can write func­tion­ing code to solve di­verse prob­lems, but the AI re­flec­tion prob­lem re­mains un­solved and thus limits the depth of re­cur­sion at­tain­able by the AIs? Let’s pro­vi­sion­ally call such gen­eral but re­flec­tion-limited AIs semi-gen­eral AIs, or SGAIs. SGAIs might be of roughly smart-an­i­mal-level in­tel­li­gence, e.g. have rudi­men­tary com­mu­ni­ca­tion/​ne­go­ti­a­tion abil­ities and some level of abil­ity to for­mu­late nar­row­ish plans of the sort that don’t leave them sus­cep­ti­ble to Pas­calian self-de­struc­tion or wire­head­ing or the like.

At first blush, this sce­nario strikes me as Bad; AIs could take over all com­put­ers con­nected to the in­ter­net, to­tally mess­ing stuff up as their goals/​sub­goals mu­tate and adapt to cir­cum­vent wire­head­ing se­lec­tion pres­sures, with­out be­ing able to reach gen­eral in­tel­li­gence. AIs might or might not co­op­er­ate with hu­mans in such a sce­nario. I imag­ine any de­tailed ex­ist­ing liter­a­ture on this sub­ject would fo­cus on com­puter se­cu­rity and in­tel­li­gent com­puter “viruses”; does such liter­a­ture ex­ist, any­where?

I have var­i­ous ques­tions about this sce­nario, in­clud­ing:

  • How quickly should one ex­pect temetic se­lec­tive sweeps to reach ~99% fix­a­tion?

  • To what ex­tent should SGAIs be ex­pected to co­op­er­ate with hu­mans in such a sce­nario? Would SGAIs be able to make plans that in­volve ex­change of cur­rency, even if they don’t un­der­stand what cur­rency is or how ex­actly it works? What do hu­mans have to offer SGAIs?

  • How con­fi­dent can we be that SGAIs will or won’t have enough oomph to FOOM once they sat­u­rate and op­ti­mize/​cor­rupt all ex­ist­ing com­put­ing hard­ware?

  • As­sum­ing such a sce­nario doesn’t im­me­di­ately lead to a FOOM sce­nario, how bad is it? To what ex­tent is its bad­ness con­tin­gent on the ca­pa­bil­ity/​will­ing­ness of SGAIs to play nice with hu­mans?

Those are the ques­tions that im­me­di­ately spring to mind, but I’d like to see who else has thought about this and what they’ve already con­sid­ered be­fore I cover too much ground.
My in­tu­ition says that think­ing about SGAIs in terms of pop­u­la­tion ge­net­ics and microe­co­nomics will some­what coun­ter­act au­to­matic ten­den­cies to imag­ine cool sto­ries rather than en­gage in dis­pas­sion­ate anal­y­sis. I’d like other sug­ges­tions for how to achieve that goal.
I’m con­fused that I don’t see peo­ple talk­ing about this sce­nario very much; why is that? Why isn’t it the de­fault ex­pected sce­nario among fu­tur­ol­o­gists? Or have I just not paid close enough at­ten­tion? Is there already a name for this class of AIs? Is the name bet­ter than “semi-gen­eral AIs”?
Thanks for any sug­ges­tions/​thoughts, and my apolo­gies if this has already been dis­cussed at length on LessWrong.